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Abstract 


The  purpose  of  this  research  was  to  determine  if  an 
analysis  method  could  be  developed  for  the  Civil  Engineering 
Environmental  Planning  Function.  The  modeling  technique 
chosen  for  this  effort  was  Constrained  Facet  Analysis  ( CFA) 
which  is  capable  of  evaluating  an  organization  based  on  a 
set  of  multiple  inputs  and  outputs.  The  most  time-consuming 
aspect  of  the  research  was  determining  appropriate  input  and 
output  measures  for  environmental  planning  organizations, 
since  the  functions  are  service-oriented  and  have  few 
variables  which  are  suitable  for  typical  quantitative 
analysis.  Many  of  the  variables  selected  were  considered 
"result"  oriented  in  that  they  measured  the  effectiveness  of 
functions  in  meeting  compliance  requirements.  The  model 
evaluated  the  measures  to  identify  efficiency  ratings  for 
environmental  planning  organizations  and  identified  which 
variables  caused  the  inefficiencies.  The  analysis  was 
accomplished  by  collecting  data  from  base  environmental 
functions  and  processing  the  data  using  the  computerized  CFA 
model.  Productivity  Assessment  Support  System.  The  results 
showed  that,  with  certain  limitations,  the  measures  and  CFA 
could  be  used  to  model  the  performance  of  environmental 
organizations.  Field  study  of  CFA  use  for  Air  Force  Civil 
Engineering  was  recommended. 
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MEASUREMENT  OF  AIR  FORCE  ENVIRONMENTAL 
PROTECTION  PROGRAM  PERFORMANCE 

I.  Introduction 

General  Issue 

Starting  in  the  late  1970 's,  the  Executive  Branch 
placed  increased  emphasis  on  improving  the  efficiencies  of 
government  agencies.  In  1982,  the  Air  Force  responded  by 
establishing  a  functional  review  program  to  evaluate 
organizational  performance.  The  Air  Force  Engineering  and 
Services  Center  (AFESC)  assumed  responsibility  for  the 
functional  review  of  Engineering  and  Services  organizations. 
The  effort  was  named  Project  IMAGE  (Innovative  Management 
Achieves  Greater  Effectiveness)  to  reflect  the  need  to 
improve  not  only  efficiency  but  overall  performance.  AFESC 
initiated  Project  IMAGE  by  holding  a  series  of  workshops  for 
functional  managers  and  conducting  or  sponsoring 
organizational  reviews. 

In  conjunction  with  AFESC' s  Project  IMAGE,  several  Air 
Force  Institute  of  Technology  students  studied  the 
performance  of  Air  Force  Civil  Engineering  organizations. 
Byers  and  Way Lett  (11)  specifically  referred  to  the  need  for 
additional  performance  reviews  of  other  Civil  Engineering 
functional  areas.  The  Air  Force  environmental  protection 
program  is  one  area  which  has  not  had  a  performance 
evaluation.  In  general,  the  environmental  program  has  not 
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been  the  subject  of  many  management  studies  due,  in  part,  to 
its  relative  newness.  Because  of  the  critical  nature  of 
environmental  affairs,  the  Air  Force  can  benefit  by  better 
understanding  the  environmental  program  and  its  responsible 
organizations . 


Problem  Statement 

The  Civil  Engineering  Environmental  and  Contract 
Planning  Organization  has  primary  responsibility  for  the 
environmental  program  at  most  bases.  The  AFESC  functional 
review  workshop  for  this  organization  was  held  10-14  May 
1986.  The  workshop  participants  developed  a  performance 
work  statement  and  a  quality  assurance  plan,  and  identified 
an  exhaustive  list  of  organizational  tasks.  However,  the 
group  did  not  identify  input  and  output  measures  and  a  model 
which  could  use  the  measures  to  adequately  evaluate  organi¬ 
zational  performance.  The  specific  problem  of  this  research 
is:  ,How  can  performance  of  the  environmental  protection 
program  best  be  evaluated,  and  how  can  performance  evalua¬ 
tion  methods  best  be  used  by  Air  Force  operational  units? 


Objectives 


The  objectives  of  this  research  are  as  follows: 

1.  Determine  appropriate  input  and  output  measures 
which  define  the  operations  of  the  environmental  protection 


function. 


2.  Collect  data  on  the  measures  from  Air  Force  Civil 


Engineering  functions. 

3.  Analyze  the  data  using  a  measurement  technique 
such  as  ratio  analysis,  linear  regression.  Data  Envelopment 
Analysis,  or  Constrained  Facet  Analysis. 

4.  Evaluate  the  measures  and  analysis  method. 

5.  Discuss  the  feasibility  of  the  practical 
application  of  this  technique. 

Investigative  Questions 

In  the  pursuit  of  investigating  the  specified  problem, 
several  investigative  questions  need  to  be  answered: 

1.  What  characteristics  best  describe  the 
environmental  protection  program? 

a.  What  are  the  environmental  program  structures 
and  what  are  the  functions  which  transform 
inputs  to  outputs? 

b.  Are  there  similar  programs/organizations  in 
the  Department  of  Defense  or  private  sector 
which  can  help  describe  these  characteristics? 

2.  What  performance  measurement  and  data  analysis 
methods  are  best  suited  for  evaluating  the 
program? 

3.  How  can  the  program  characteristics  be  described 
in  terms  of  program  inputs  and  outputs  for 
performance  analysis? 


4. 


How  can  environmental  managers  use  these  study 
results  and  recommendations  to  evaluate  and  improve 
performance? 

5.  What  procedures  (e.g.,  computer  models)  can  be 
recommended  for  use  of  the  selected  analysis 
method? 

6.  What  recommendations  for  operational  use  or 
improvement  do  analysis  results  suggest? 


Operational  Definitions 

In  answering  the  research  question  and  evaluating  the 

performance  of  the  Air  Force  environmental  program,  the 

following  operational  definitions  will  be  used: 

Performance  -  overall  manner  in  which  an  organization 
fulfills  its  intended  purpose  as  measured  against  some 
standard. 

Productivity  -  a  combination  of  efficiency  and 
effectiveness . 

Efficiency  -  producing  the  greatest  results  with  a 
given  set  of  resources  (measured  in  terms  of 
output/ input ) . 

Effectiveness  -  the  ability  to  accomplish  the  correct 
organizational  tasks. 

Input  -  resources  going  into  an  organization. 

Output  -  products  or  services  of  an  organization. 

Environmental  Planning  Function  (EPF)  -  the  function, 
usually  located  in  Environmental  and  Contract  Planning, 
which  has  primary  responsibility  for  the  base 
environmental  protection  program. 

Decision  Making  Unit  (DMU)  -  the  organizational 
element,  in  this  case  the  EPF,  which  will  be  analyzed. 
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Scope  and  Limitations 

The  scope  of  this  research  is  limited  to  investigating 
and  proposing  methods  for  managers  to  evaluate  the  perfor¬ 
mance  of  Air  Force  environmental  programs.  No  base-specific 
recommendations  for  program  or  organizational  changes  will 
be  made.  Therefore,  these  research  results  should  be  used 
as  techniques  to  evaluate  organizational  productivity,  not 
to  penalize  or  threaten  organizations. 

Assumptions  of  the  research  are: 

1.  The  input  and  output  data  supplied  by  the  Air  Force 
Civil  Engineering  organizations  are  valid  and  accurate. 

2.  The  ultimate  goal  of  the  environmental  program  is  to 
protect  the  environment  and  ensure  regulatory  compliance. 
Effectiveness  in  striving  for  this  goal  should  be  a  higher 
priority  than  efficiency  or  serious  consequences  for  the 
base  and  surrounding  community  could  result. 

3.  The  magnitude  of  environmental  requirements  have  grown 
dramatically  over  the  last  five  years.  Bases  are  at  various 
stages  in  making  personnel  and  budget  adjustments  to  meet 
these  new  requirements. 

4.  The  population  of  interest  will  not  include  overseas 
bases.  Air  Force  Reserve  units  or  Air  National  Guard  units, 
since  their  programs  are  not  comparable  in  size  and  other 
ways  to  most  Continental  United  States  (CONUS)  bases' 
programs.  Excluding  these  units  does  not  preclude  their 
benefiting  from  future  application  of  this  research. 


II 


Literature  Review 


Background 

In  the  last  two  decades,  productivity  has  become  a 
growing  concern  for  most  organizations  in  the  United  States. 
Productivity  growth  has  declined  during  this  period,  with 
the  growth  rate  decrease  starting  in  1968  and  turning  nega¬ 
tive  in  1979  (40:28).  Even  with  United  States  trends  back 
to  positive  growth  starting  in  1981,  nations  such  as  Canada, 
West  Germany,  and  Japan  could  surpass  our  total  productivity 
by  the  1990’s  (1:6;  40:28).  A  reversal  in  productivity 
trends  is  essential  to  avoid  a  severe  decline  in  our 
economic  strength  and  worldwide  political  influence  (40:28). 

Several  factors  contribute  to  the  national  decline  in 
productivity.  A  major  reason  is  the  reduction  in  this 
country's  market  share  of  technological  innovations  from  80 
to  50  percent  (1:5).  This  decline  results  in  part  from  our 
inability  to  employ  enough  qualified  technical  personnel  in 
positions  capable  of  keeping  abreast  of  new  technology 
(1:7).  A  steady  shift  from  a  blue-collar  to  a  white-collar 
workforce  has  also  altered  the  productivity  growth  rate 
which  is  measured  significantly  by  industrial  production 
(20:27).  Other  reasons  for  more  productivity  decline 
include  high  labor  costs  with  no  corresponding  productivity 


improvement,  aging  factories,  increased  emphasis  on  services 
rather  than  industrial  production,  and  high  unemployment 
rates  (9:5)  . 


Performance  Concepts 


Before  considering  the  significance  of  performance  to 

the  Air  Force,  key  terms  and  concepts  of  performance 

measurement  need  to  be  explored.  The  following  discussion 

should  clarify  the  ideas  that  form  the  basis  of  the 

operational  definitions  stated  in  Chapter  I. 

DOD  Instruction  5010.34  provides  definitions  of 

productivity,  effectiveness,  and  efficiency  for  use  in 

evaluating  overall  performance  in  DOD  agencies. 

Productivity:  The  efficiency  with  which  organizations 
utilize  all  types  of  fund  resources  (operating  and 
investment)  to  accomplish  their  mission  represents 
total  resource  productivity.  The  efficiency  with  which 
organizations  utilize  labor  resources  to  accomplish 
their  missions  represents  labor  productivity.  (21:1) 

Efficiency:  Efficient  means  accomplishing  the  right 

things  with  the  lowest  possible  expenditure  of 
resources.  (21:1) 

Effectiveness:  Effectiveness  means  accomplishing  the 

right  things  in  the  right  quantities,  at  the  right 
times.  (21:1) 

Productivity  depends  on  both  effectiveness  and  efficiency. 

In  simple  terms,  it  is  not  only  doing  the  right  things,  but 
doing  them  right  (9:10).  Efficiency  is  a  ratio  of  outputs 
to  inputs.  It  is  regarded  as  the  "quantity  factor  of 
productivity"  and  as  "the  amount  of  output  per  unit  of  input 
without  regard  to  organizational  goals  or  objectives" 

(23:17).  Effectiveness  can  be  defined  as  the  relationship 
between  work  performed  and  organizational  objectives 
(23:18).  It  is  the  quality  of  having  a  desired  effect 


Critical  to  assessing  organizational  productivity  is 
the  need  to  describe  measurable  inputs  and  outputs.  Input, 


or  resources  consumed,  is  usually  expressed  in  terms  of 
dollars  or  manhours  and  can  be  relatively  easily  quantified 
(1:5).  Output,  the  products  or  services  of  an  organization 
is  not  so  easily  quantified.  At  least  products  have  charac 
teristics  which  can  aid  in  their  measurement:  a)  Products 
tend  to  remain  unchanged  in  contrast  to  people  or  organiza* 
tional  arrangements,  so  their  cost  can  be  described  with  a 
degree  of  certainty;  and  b)  Products  are  usually  described 
in  a  common  language,  so  they  are  useful  concepts  when 
aggregating  costs  (30:48). 

The  concept  of  productivity  has  expanded  to  include 
not  only  the  quantity,  but  the  quality  of  output.  In  more 
recent  times,  the  emphasis  has  switched  from  efficiency  to 
effectiveness.  Organizations  are  now  considered  "social 
entities"  responsible  for  quality  products,  environmental 
protection,  and  work  place  conditions  (32:11).  Including 
this  quality  concept  further  complicates  attempts  to 
quantitatively  measure  productivity  (1:5). 

Improving  Productivity 

Improving  productivity  is  one  of  the  few  realistic 
means  of  maintaining  growth  for  public  sector  organizations 
such  as  the  DOD.  With  current  budget  and  manpower  con- 
straints  often  coupled  with  recession  and  inflation,  DOD 
cannot  just  raise  production  to  keep  up  with  increasing 


demands.  A  key  part  of  the  solution  is  to  get  more  accom* 
plished  with  available  resources;  thus,  productivity,  not 
production  is  the  main  issue.  "The  public  sector  must  seek 
continual  productivity  improvement  in  order  to  provide 
essential  services  at  a  minimum  cost  to  the  public"  (2:16). 
Substantial  gains  in  productivity  are  feasible  in  service 
oriented  organizations  which  are  inherently  labor  intensive 
(40:28) . 

Productivity  enhancement  efforts  are  not  recent 
developments  in  DOD.  For  example,  as  early  as  1975,  DOD 
established  the  "Defense  Productivity  Program  in  order  to 
bring  productivity  considerations  into  the  mainstream  of 
defense  management"  (20:27).  DOD  Instruction  5010.34 
<4  August  1975)  and  DOD  Directive  5010.31  (27  April  1979) 
were  published  so  that  each  component  would  develop  methods 
to  improve  productivity  (11:15*16).  .  Over  a  billion  dollars 
are  invested  annually  in  salaries  and  expenses  for  ongoing 
productivity  enhancement  programs  (2:13).  Rather  than 
representing  a  duplication  of  programs,  the  current 
efficiency  review  approach  "serves  as  an  integrating 
framework  for  the  various  other  efforts"  (2:14). 

Office  of  Management  and  Budget  Circular  A*76 
established  federal  policy  that  the  government  shall  rely  on 
the  private  sector  to  provide  goods  and  services  to  the 
greatest  extent  possible.  An  activity  can  be  contracted  out 
to  private  concerns  if  it  is  not  a  restricted  "governmental 


function"  and  if  a  cost  comparison  shows  the  private  firm  to 
be  less  costly.  In  1981,  DOD  directed  each  branch  of 
service  to  establish  formal  procedures  for  efficiency 
reviews  of  activities  not  subject  to  contracting  out  under 
A-76  provisions.  The  A-76  process  requires  the  services  to 
prepare  performance  work  statements  which  describe  work  to 
be  accomplished  and  associated  performance  standards. 

Besides  getting  managers  to  critically  assess  organizational 
goals,  functions,  and  standards,  "performance  work  state¬ 
ments  act  as  a  catalyst  for  new  ideas  regarding  work 
performance"  (2:14).  Support  for  this  approach  has  been 
proffered  by  the  Government  Accounting  Office  which  foresees 
additional  savings  possibilities  and  the  private  sector 
which  anticipates  more  contracting  opportunities  (2:14). 

Under  the  1981  productivity  enhancement  directive, 
each  branch  of  service  was  allowed  to  develop  its  own 
approach  to  improving  productivity.  The  Army's  initial 
effort  was  a  decentralized  approach  with  two-man  teams 
assigned  to  major  units.  The  teams  applied  various 
improvement  techniques  including  team-building  concepts, 
goal-setting  ideas,  and  survey-guided  development  (20:28). 

The  Navy  developed  a  survey-guided  system  with  teams 
of  experienced  personnel  operating  out  of  five  consulting 
centers.  Team  members  surveyed  communication  flow, 
decision-making  processes,  supervisory  and  peer  leadership, 
and  work  group  processes;  then  made  recommendations  for  each 
area  (20:28). 
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The  Air  Force  formed  the  Air  Force  Leadership  and 
Management  Development  Center  staffed  by  military  personnel 
displaying  exceptional  leadership  abilities.  Teams  from  the 
Center  visited  bases  on  a  routine  basis  to  identify  manage¬ 
ment  concerns  and  to  make  recommendations  to  improve  produc¬ 
tivity  through  different  leadership  and  management  techni¬ 
ques.  The  teams  performed  follow-up  visits  to  evaluate 
implementation  success  (20:29). 

Performance  improvement  programs  have  been  relatively 
well  established  for  blue-collar,  profit-oriented  organiza¬ 
tions.  As  a  result  productivity  gains  have  been  concen¬ 
trated  in  non-management,  non-professional  staffs.  For 
example,  44  data  transcribers  at  McClellan  Air  Force  Base 
saved  the  Air  Force  $26,000  in  21  weeks  of  a  pay  incentive 
program  (20:30).  These  administrative  personnel  perform 
"functions"  which  are  relatively  easy  to  describe  in 
productivity  terms.  In  contrast  to  "tasks"  performed  by 
professionals,  "functions"  are  easier  to  manipulate  (e.g., 
automate)  and  thereby  have  more  direct  influence  on 
productivity  (30:48). 


Measurement  Difficulties  in  Publ ic  Agencies 

Even  with  the  emphasis  on  performance  enhancement,  DOD 
takes  a  deliberate  and  cautious  approach  to  making  organiza¬ 
tional  changes  for  productivity's  sake.  Attempts  to  improve 
productivity,  often  done  through  increasing  automation,  have 
been  counterproductive  without  properly  evaluating 


organizations  prior  to  implementation  (30:47).  Requisites 

of  a  successful  program  are  comparative  standards  and 

productivity  measurement  methods  (9:9;  11:16),  but  these 

are  not  well  established  in  the  government  (11:11). 

Methods  for  evaluating  the  relative  productivity  of 
decision  making  units  in  the  public  sector  have  lagged 
behind  similar  applicati  ns  where  production  functions 
were  more  directly  obtainable.  (5:57) 

The  measurement  difficulty  is  exacerbated  when  attempting  to 

use  quantitative  measurement  techniques  since  public 

organizations  cannot  always  use  a  standard  evaluation 

factor,  such  as  dollars,  for  input  and  output.  To  overcome 

this  deficiency,  public  agencies  have  tried  at  least  two 

approaches:  a)  incorporating  appropriate  qualitative 

methods  to  supplement  existing  quantitative  measures  and  b) 

improving  or  increasing  the  quantitative  methods  appropriate 

for  use  by  nonprofit  organizations.  The  remainder  of  this 

chapter  reviews  the  problems  that  government  agencies 

encounter  in  measuring  productivity,  then  discusses  how 

improvements  can  be  made  using  the  two  methods  mentioned 

above . 

Anthony  and  Young  (3:40)  elaborate  on  the  classic 
general  difficulties  in  measuring  public  agency  performance. 
The  availability  of  a  profit  measure  is  probably  the  key 
factor  in  how  effectiveness  criteria  are  established  in 
organizations.  In  a  prof it*or iented  company,  success  is 
measured  by  the  amount  of  profits  earned,  and  managers  make 
decisions  on  which  alternatives  produce  the  largest  profit 
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margin.  Since  military  organizations  are  service-oriented 
and  profit  is  not  a  primary  objective,  management  decisions 
are  made  on  the  basis  of  providing  the  best  possible  service 
with  available  resources  (3:35).  Thus,  the  success  of  a  DOD 
organization  is  often  hard  to  determine  without  profit 
measures.  Additional  measurement  problems  arise  because 
success  is  often  considered  to  be  related  to  the  size  of 
agency  budgets  and  performance  data  is  seldom  collected  in 
usable  form  (31:159). 

Research  focused  on  mental  health  organizations,  law 
enforcement  agencies,  and  regulatory  agencies  identified 
several  specific  impediments  to  ensuring  and  assessing 
productivity  in  the  nonprofit  public  agency  environment. 

1.  The  boundaries  of  a  public  mental  health  agency's 
responsibilities  are  not  well  defined,  and  the  agency 
usually  does  not  control  all  mental  health  activities  within 
its  boundaries  (4:192). 

2.  The  environment  in  which  public  agencies  operate  is 
characterized  by  conflicting  public  expectations.  Mental 
health  agencies  commonly  do  not  have  a  publicly  recognized 
set  of  outcomes  or  a  consensus  on  criteria  to  use  in 
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measuring  productivity  (4).  Environmental  agency  decision¬ 
making  also  must  deal  with  divergent  public  input,  as 
exemplified  by  public  comments  on  Environmental  Impact 
Statements.  Even  though  public  pressure  can  affect  agency 
actions,  the  agency  stands  alone  when  its  performance  is 
evaluated . 
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3.  Many  public  sector  organizations  have  poorly 
defined  programs  or  complex  tasks  to  perform.  Public  health 
agencies  have  difficulties  in  specifying  the  optimal  type 
and  amount  of  therapy  for  patient  symptoms  (4).  A  single 
law  enforcement  case  can  involve  thousands  of  bits  of 
evidence  and  can  last  for  years  with  concurrent  personnel 
loses  (16:210).  While  most  forms  of  pollution  (e.g., 
chemical  spills)  are  obvious,  the  consequences  and  solutions 
for  environmental  agencies  are  not  always  so  clear.  The 
enforcement  process  for  environmental  violations  is  often 
time-consuming  and  cumbersome,  involving  lengthy  court  cases 
with  complicated  evidence  (35:170;  36:233). 

4.  Many  agencies'  roles  are  complicated  by  relation¬ 
ships  with  organizations  at  other  levels  of  government. 
Federal  law  enforcement  agencies  have  many  options  in 
supplementing  state  and  local  police  even  though  a  case  may 
not  be  federal  responsibility.  Federal  environmental 
agencies  rely  on  states  for  support,  but  success  may  reflect 
a  state's  regulatory  philosophy  more  than  the  federal 
program's  effectiveness. 

5.  A  public  agency  competes  with  other  agencies  for 
limited  funds,  and  various  units  within  the  agency  compete 
for  agency  funds.  If  Congress  stops  funds  for  certain 
programs,  overall  agency  goals  can  be  hurt.  When  several 
federal  agencies  were  responsible  for  environmental  protec¬ 
tion,  Congress  failed  to  commit  adequate  pollution  control 
funds  because  of  the  diverse  agency  interests  (35:140). 


Measurement  Incorporating  Qualitative  Methods 

In  the  1960's  and  1970's,  federal  agencies  tended  to 
rely  too  much  on  quantitative  measurement  and  downplayed 
qualitative  matters  which  contributed  to  overall  performance 
(14:).  The  system  was  popular  for  many  reasons  in  that 
( 16:214) : 

1.  It  was  consistent  with  prevailing  systems  which 
stressed  computerized  manipulation  of  "hard  data." 

2.  The  data  allowed  program  comparisons  using 
increasingly  popular  cost-benefit  analyses. 

3.  Managers  could  rationalize  resource  allocation 
decisions  by  citing  data  analyses. 

4.  It  indicated  which  agency  effort  to  emphasize  by 
showing  which  had  the  best  "return  for  the  dollar." 

The  Federal  Bureau  of  Investigation  realized  that 
quantification  was  being  overdone  and  recognized  the 
following  as  the  system's  limitations  (16:215): 

1.  Insensitivity  to  qualitative  considerations  such  as 
whether  a  deadly  weapon  was  used  in  committing  a  crime. 

2.  Encouraging  suboptimization  by  overstating  less 
important  crimes  merely  to  show  numerical  success. 

3.  Tendency  of  managers  to  respond  to  an  artificial 
"bottom  line"  created  by  the  quantitative  reports. 

Many  federal  agencies  started  using  more  "performance 


evaluations"  after  realizing  that  qualitative  measures  can 
be  used  to  supplement  quantitative  measures  to  better 


reflect  overall  organization  performance.  Performance 
evaluations  are  broader  in  concept  than  most  measurement 
methods  because  they  describe  overall  system  functioning 
(i.e.,  the  manner  in  which  a  system  accomplishes  results) 
while  taking  into  account  efficiency,  effectiveness,  and 
productivity. 

The  evaluations  better  assess  the  quality  of  service  in 
an  area  concerned  with  human  values.  On  the  other  hand, 
quality  usually  can  be  secondary  to  profit  or  some  other 
"bottom  line"  measurement  in  the  private  sector.  Performance 
evaluation  is  important  for  all  organizations  to  assist  in 
tactical  and  strategic  planning,  but  is  necessary  for  public 
organizations  where  efficiency  ratings  cannot  be  used  as 
sole  indicators  of  organizational  health  (14). 

By  the  mid*1970's  the  Federal  Bureau  of  Investigation 
had  modified  its  evaluation  system  to  include  subjective 
measures  such  as  inspections  by  independent  auditors,  field 
management  assessments,  and  end»user  surveys.  The  system 
evolved  from  a  periodic  event  into  more  of  a  reporting 
process,  where  many  measures  are  evaluated  on  a  continuing 
basis  (16:217*220) . 

Many  other  federal  agencies  have  attempted  to  maintain 
a  balance  of  objective  and  subjective  measures  in  their 
programs.  For  example,  the  Air  Force  includes  Management 
Effectiveness  Inspections,  compliance  reviews,  and  indepen* 
dent  audits  to  supplement  its  array  of  data  analysis 


reports.  For  social  service  agencies,  the  implementation  of 
Title  XX  of  the  1975  Social  Security  Act  fostered  the 
development  of  a  performance  assessment  system  which 
combines  "quantitative,  objective  data  with  more  subjective 
procedures  for  judging  the  quality  of  service"  (10:270). 

And  many  health  agencies  consider  the  subjective  measure  of 
user  (i.e.,  patients)  satisfaction  an  excellent  indicator  of 
performance  (4:204). 

Measurement  Using  Improved  Quantitative  Methods 

While  performance  evaluation  is  a  worthy  concept 
applicable  to  public  agencies,  there  are  other  ways  to 
improve  performance  measurement.  Performance  evaluation 
relies  on  managers  to  develop  appropriate  measures  which 
reflect  organizational  activity.  If  these  measures  can  be 
used  as  inputs  or  outputs  for  quantitative  analysis  techni¬ 
ques,  then  the  manager's  job  of  performance  measurement  is 
made  easier.  Being  able  to  quantify  and  evaluate  this  type 
of  variable  requires  that  agencies  obtain  better  analysis 
techniques. Performance  standards  have  been  developed  for  DOD 
operational  units  which  are  characteristically  blue-collar, 
profit-oriented  organizations.  Few  performance  criteria 
exist  for  governmental  staff  organizations  such  as  legal  or 
other  professional  offices.  The  importance  of  staff  offices 
increases  as  our  society  becomes  more  service  and  informa¬ 
tion  processing  oriented.  The  demand  for  DOD  staff  func¬ 
tions,  such  as  preparing  legal  briefs  and  environmental 


assessments,  has  grown  due  to  greater  technological  sophis* 
tication  and  more  complex  regulations  (37:17*18).  DOD 
productivity  is  not  accurately  portrayed  due  to  the  over* 
emphasis  on  blue*collar  measurement  while  the  white*collar 
support  roles  are  rapidly  growing  (37:18). 

In  1980,  DOD  tasked  the  Navy  to  develop  productivity 
measures  for  staff  functions.  Determining  that  there  was  no 
straightforward  method  as  used  for  operational  units,  a  Navy 
task  group  developed  a  multi*step  approach  based  on 
"variance  technique"  (37:18).  Thirty  items  of  performance 
deviation  in  nine  key  areas  (e.g.,  "fuzzy  or  unclear 
tasking")  formed  the  basis  for  the  performance  standards. 
Staff  performance  was  then  evaluated  by  how  much  operations 
deviated  from  the  ideal.  Besides  providing  the  benefit  of 
better  identifying  individual  job  requirements  and  organiza¬ 
tional  performance  goals,  the  project  demonstrated  the 
feasibility  of  developing  staff  productivity  measures. 
However,  there  are  limitations  to  the  study  since  the 
measures'  "appropriateness  and  validity  within  any  one  unit 
or  across  multiple  staff  units  need  further  investigation 
and  testing"  (37:21). 

The  Navy  has  sponsored  research  into  other  areas 
involving  measuring  productivity  of  staff  organizational 
functions.  The  "current  value  human  resources  accounting" 
approach  focuses  on  the  relationship  between  the  current 
state  of  the  human  organization  and  future  performance.  Its 


goal  is  to  aid  decision  making  by  providing  information 
about  the  effects  of  organizational  policies  on  the  value  of 
human  resources  (32:185).  While  it  does  not  establish  a 
directly  applicable  methodology,  the  study  provides  insight 
into  performance  measurement  of  relatively  qualitative 
functions.  For  example,  some  output  measures  (e.g., 
operational  readiness)  cannot  be  easily  quantified  into 
monetary  terms.  The  study  argues  for  "analogues  to 
commercial -world  accounting  notions"  which  could  apply  to 
DOD  (32:172).  Most  importantly,  the  researchers  showed  that 
a  system  which  worked  in  the  private  sector  could  be  applied 
to  a  public  sector  organization,  the  Navy. 

As  part  of  its  productivity  enhancement  program,  the 
Air  Force  is  working  to  accomplish  performance  evaluations 
of  all  functional  areas.  The  Air  Force  Institute  of 
Technology  is  assisting  in  this  effort  by  using  reviews  as 
thesis  projects.  Theses  by  Donovan  (22)  and  Hitt  and  Horace 
(28)  assessed  the  productivity  of  depot-level  maintenance 
operations  at  Newark  Air  Force  Station  and  in  the  Air  Force 
Logistics  Command  respectively.  The  Major  Command  requested 
Donovans' s  research  in  order  to  find  a  model  to  measure 
"total  factor  technical  productivity"  which  could  be  applied 
command-wide  (22:6). 

These  research  efforts  involved  measures  of  relatively 
quantifiable  data,  but  other  theses  have  been  completed  in 
areas  which  are  more  difficult  to  measure.  Fisher  (23) 
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evaluated  the  Civil  Engineering  organization  which  is 
service-oriented  and  usually  more  difficult  to  quantify  than 
maintenance  operations.  Byers  and  Waylett  emphasized  the 
difficulty  in  defining  public  sector  productivity, 
especially  for  military  fire  departments  (11:17).  All  of 
these  efforts  relied  on  analytical  tools.  Data  Envelopment 
Analysis  and  its  successor  Constrained  Facet  Analysis,  which 
have  been  shown  to  be  effective  in  evaluating  qualitative 
data  found  in  staff  functions. 

Date  Envelopment  Analysis  (DEA)  was  developed 
specifically  for  not-for-profit  organizations  where  it  is 
important  to  evaluate  "program  efficiency"  and  "managerial 
efficiency"  (5:57).  The  early  use  of  DEA  was  to  evaluate 
the  productivity  of  schools.  In  contrast  to  previous 
regression  methods  which  compared  the  relative  effects  of 
variables  on  achievement,  DEA  compared  school  units  relative 
to  their  input  and  output  efficiencies.  Using  DEA, 
researchers  were  able  to  identify  1)  inefficient  school 
units;  2)  input  and  output  variables  contributing  to 
inefficiency;  and  3)  changes  in  the  inputs  and  outputs 
required  to  make  the  unit  efficient  (5:71).  The  inability 
of  DEA  to  provide  planning  information  prompted  the 
formulation  of  a  new  method  of  calculating  efficiency  called 
Constrained  Facet  Analysis  (CFA)  (22:8). 


Successful  Application  of  Productivity  Programs 

Two  elements  appear  to  dramatically  affect  the  success 
of  a  performance  enhancement  and  evaluation  programs- -people 
and  perseverance.  Regardless  of  which  approaches  are  used 
to  measure  and  improve  performance,  people  are  essential  to 
implementation.  Performance  gains  result  only  from  changes 
made  to  individuals  and  organizations  (9:9).  While  produc¬ 
tivity  strategies  may  "draw  the  distinction  between  manage¬ 
ment  productivity  and  that  being  managed,"  all  levels  of 
personnel  can  benefit  (40:32). 

Research  and  experience  both  demonstrate  that  improved 
productivity  and  performance  depend  on  not  only 
workers'  attitudes  and  abilities  but  to  a  great  extent 
on  the  attitudes  and  abilities  of  management  as  well. 
(20:31) 

An  organization  must  persevere  to  keep  from  losing 
productivity  and  individual  involvement  (9:14).  To  sustain 
this  drive,  organizational  goals  and  objectives  may  need  to 
be  modified  to  reflect  environmental  changes;  therefore,  the 
performance  model  should  be  periodically  run  to  make  new 
assessments  of  changed  measures.  Circumstances  may  vary, 
but  the  results  of  these  evaluations  and  the  objectives  to 
be  reached  usually  should  be  shared  with  the  keystone  of  the 
organization- -  individuals . 

The  path  for  future  performance  evaluation  efforts 
appears  to  lead  toward  increased  analysis  of  staff  organiza¬ 
tions  in  the  public  sector.  Analytical  models,  such  as  DEA 
or  CFA,  have  successfully  been  applied  to  situations  where 
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traditional  productivity  methodologies  were  unsuccessful. 
Even  though  approaches  to  measuring  and  improving  perfor¬ 
mance  in  the  public  and  private  sectors  may  differ,  it  is 
important  that  the  government  learn  from  successful  commer¬ 
cial  efforts.  In  this  regard,  DOD  has  studied  private 
sector  performance  incentive  plans  with  the  objective  of 
determining  applicability  to  the  services  (20:31).  The 
benefit  was  primarily  for  blue-collar  organizations,  but 
similar  efforts  should  continue  for  white-collar 
organizations . 

The  Air  Force’s  current  review  of  functional  areas  has 
several  ancillary  benefits. 

1.  It  assists  in  quantifying  staff  organizations' 
inputs  and  outputs  and  defining  job  requirements  or 
organizational  goals  (37:21). 

2.  It  continues  to  validate  DEA  and  CFA  as  effective 
analytical  tools. 

3.  It  provides  investigation  into  private/public  ' 
sector  measurement  and  techniques. 

Finding  a  methodology  that  is  not  only  effective  but 
easily  implemented  by  base  level  units  is  important,  since 
periodic  evaluations  are  required  as  program  changes  are 
made  (9;  20).  The  results  then  should  be  made  known  to 
affected  staffs,  and  comparison  of  relative  performance 
between  units  may  be  an  important  reinforcement  technique. 
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Conclusion 


Considering  the  potential  benefits,  a  review  of  the 
performance  of  Air  Force  environmental  protection  functions 
should  be  of  considerable  benefit.  At  a  minimum,  the  func- 
tion  can  benefit  from  an  evaluation  of  inputs,  outputs,  and 
program  objectives,  since  these  elements  of  the  environ¬ 
mental  program  traditionally  have  not  been  well  quantified. 
As  the  price  for  not  complying  with  environmental  regula¬ 
tions  rises,  base  civil  engineers  and  commanders  should  be 
increasingly  concerned  with  program  performance.  This  is 
particularly  true  because  the  public  now  considers  busi¬ 
nesses  and  public  organizations  to  be  "social  entities"  with 
responsibilities  for  protecting  the  public  and  the 
environment  (1:5) 
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III.  Developing  Public  Agency  Performance  Measures 
Introduction 

As  discussed  in  Chapter  II,  public  and  service- 
oriented  organizations  are  more  difficult  to  evaluate  than 
private  and  product-oriented  organizations.  A  key  problem 
is  defining  the  organization  in  terms  of  input  and  output 
measures,  which  depends  on  a  thorough  understanding  of  the 
organization.  Assuming  that  some  of  these  same  difficulties 
apply  to  the  Environmental  Planning  Function  (EPF),  the 
experiences  of  public  agencies  or  other  service-oriented 
organizations  could  be  used  to  determine  better  ways  to 
evaluate  the  function.  This  chapter  answers  the  first  and 
part  of  the  third  investigative  questions  by  describing 
environmental  organizations  and  proposing  potential 
measures. 

To  assist  in  developing  the  measures,  the  author 
attempted  to  research  performance  measurement  accomplished 
by  other  environmental  organizations  (private  or  public) 
similar  to  the  EPF,  as  was  done  by  Byers  and  Waylett  for 
fire  departments  (11).  Unfortunately,  little  of  the 
available  literature  evaluated  environmental  organizations' 
performance.  One  reason  for  the  shortage  may  be  the 
relative  new  role  of  environmental  planning  offices  in 
business  and  government.  Second,  while  engineering 
organizations  typically  measure  the  efficiency  of  production 
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or  construction  operations,  they  scrutinize  their  own 
engineering  functions,  including  environmental  planning,  to 
a  lesser  degree. 


Nonprofit,  Service-Oriented  Performance  Measures 

Difficulties  in  developing  productivity  measures  have 
been  studied  in  such  public  organizations  as  social  service 
agencies,  health  services,  mass  transit  systems,  police 
departments,  and  sanitation  services.  "The  fundamental 
problem  stems  from  the  ambiguity  of  the  tasks  involved  in 
providing  human  services  and  the  lack  of  standard  defini¬ 
tions  for  the  array  of  available  services"  (10:269).  While 
it  is  difficult  to  quantify  subjective  human  factors.  Bush 
proposes  surrogate  measures  for  evaluating  caseworker 
productivity. 

The  total  number  of  service  hours  would  be  one  quanti¬ 
tative  measure  or  estimate  of  input  towards  goal 
attainment.  Other  inputs  considered  might  include 
caseworker  salary  and  administrative  overhead  costs. 
Finally,  case  closings  might  be  used  as  an  estimate  of 
direct  output.  The  need  to  consider  direct  output 
along  with  an  estimate  of  consequences  on  an  indivi¬ 
dual  case  basis  is  obvious.  The  case  closing  could 
have  resulted  from  a  circumstance  other  than  meeting 
the  designated  goal  of  self-sufficiency.  While  not  a 
rigorous  scientific  methodology,  such  estimates  can, 
in  combination,  serve  as  indicators  of  productivity. 
(10:273) 

Tne  last  point  about  combining  measures  is  critical  in  that 
more  measures  are  often  needed  to  represent  a  subjective 
activity  than  in  an  objective  case  where  number  of  dollars 
(input)  and  number  of  widgets  (output)  can  nearly  represent 
an  entire  manufacturing  operation. 
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To  facilitate  the  evaluation  of  mental  health  service, 
Arrington  and  Biskin  (4)  add  the  measurement  category  of 
"equity"  to  the  commonly  used  efficiency  and  effectiveness. 
The  issue  of  equity  in  the  availability  of  services  focuses 
on  who  the  system  is  designed  to  service.  Difficulties  in 
developing  equity  measures  arise  since  there  is  not  always  a 
"natural  one-to-one  relationship  between  services  sought  and 
services  available"  (4:194).  Investing  in  those  patients 
with  the  greatest  need  may  not  result  in  the  most  improve¬ 
ment  and  subsequent  benefit  to  society;  therefore,  total 
number  of  patients  serviced  should  not  be  the  only  indicator 
of  agency  performance. 

In  developing  efficiency  measures  which  quantify  the 
amount  of  health  services  staff  time,  the  researchers 
distinguished  between  programs  that  required  different  staff 
members'  activities.  Total  manhours  associated  with  a 
service  cannot  be  assumed  to  represent  the  same  wages  and 
professional  training.  As  general  advice  for  developing 
measures,  the  researchers  state  that  the  effort  should  be 
restricted  "to  those  questions  about  which  reliable 
information  can  be  developed  and  in  which  there  is,  in  fact, 
management  discretion."  (4:205) 

Public  transit  is  another  field  in  which  there  has 
been  increasing  efforts  to  deal  with  performance  measure¬ 
ment.  "Unfortunately,  too  little  has  been  done  to  develop 
operational  definitions  of  performance,  or  to  identify  the 
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weaknesses  and  biases  inherent  in  certain  types  of 
performance  measures."  (25:379).  Since  transit  revenues  in 
the  form  of  subsidies  are  increasingly  being  linked  to 
performance,  system  survival  is  partially  dependent  on 
accurate  measurement. 

Two  problems  often  encountered  in  the  transit  field 
are  overlapping  efficiency  with  effectiveness  and  using 
ratios  inappropriately.  In  contrast  to  efficiency, 
effectiveness  measures  generally  should  be  "cost  free" 
(25:381).  Effectiveness  ratios  are  often  used  without 
considering  the  need  to  also  evaluate  the  overall 
effectiveness  of  the  function.  For  example,  a  bus  ridership 
ratio  may  indicate  System  A  is  "better"  than  System  B  ever, 
though  System  B  has  more  total  ridership.  The  resultant 
claim  about  System  A  might  be  misleading  about  goal 
achievement  (25:381). 

While  Gleason  and  Barnum  did  not  provide  ideas  for 
measures  specifically  applicable  to  the  EPF,  their  general 
advice  was  provided. 

The  exact  set  of  performance  measures  chosen  should  be 
based  on  objectives  of  the  use,  should  measure  all 
objectives,  and  should  include  elements  which  are 
controllable  by  the  system.  There  are  no  performance 
measures  universally  appropriate,  and  multiple 
measures  must  be  used.  (25:385) 

The  author's  review  of  the  literature  attempted  to 
examine  private  environmental  organizations  for  ideas  on 
their  measurement  efforts.  Since  no  articles  were  found, 


the  review  moved  to  the  analysis  of  engineering 
organizations.  Much  of  the  engineering  literature  focuses 
on  the  productivity  of  production  or  construction  operations 
which  have  convenient,  discrete  measurement  units  such  as 
manhours  and  bricks.  However,  the  engineering  management 
field  has  studied  areas  which  have  characteristics  similar 
to  those  of  environmental  planning  functions. 

Based  on  a  survey  of  engineering  executives,  there  is 
high  potential  for  engineering  productivity  improvement  in 
planning  and  scheduling  and  a  moderate  potential  in 
communication  and  engineering  (13:40).  Case  histories 
researched  by  Hensey  and  Gibble  (27)  generally  support  this 
survey  and  explain  how  productivity  can  be  increased  in 
these  areas  by  improving  performance  reviews  and  feedback 
mechanisms.  Identifying  where  and  how  to  make  productivity 
improvements  requires  measurement  capabilities. 

The  "hierarchy  model,"  developed  for  the  construction 
industry,  provides  measurement  capability  for  these 
engineering  management  areas.  The  model  allows  continuing 
analysis  of  construction  "productivity  flow  from  project 
conception  to  completion,"  and  addresses  all  industry 
levels,  including  policy  formation,  program  management, 
engineering,  and  site  construction  (29:138).  By  reducing  a 
construction  project  to  its  basic  components,  the  hierarchy 
model  and  its  measures  fully  describe  the  total  productivity 
expression.  Suggestions  for  measure  development  based  on 
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experience  with  the  model  include  a)  concentrating  on 
measurement  of  effectiveness  versus  efficiency,  b) 
maximizing  measurement  of  total  productivity  versus 
component  parts,  and  c)  attempting  to  use  measures 
applicable  to  similar  types  of  decision  making  units 
(29:142-143)  . 

As  the  cited  experts  have  stated,  establishing  the 
criteria,  or  measures,  by  which  performance  is  measured  is 
an  important  initial  step  in  performance  measurement.  Table 
3.1  is  a  summary  of  guidelines  for  establishing  measures 
proposed  by  several  authors  of  the  articles  which  were 
researched . 

Potential  Measures  for  Environmental  Agencies 

This  section  evaluates  regulatory  agencies,  with 
emphasis  on  the  Environmental  Protection  Agency  (EPA),  to 
determine  their  performance  characteristics.  The  analysis 
shows  similarities  between  regulatory  agencies  and  the  EPF 
in  anticipation  of  applying  some  of  these  agencies'  measures 
to  the  EPF. 

Agencies'  roles  are  to  provide  goals,  standards, 
priorities,  and  monitoring  and  administrative  systems  to 
carry  out  their  legislated  purpose.  They  can  use  economic 
incentives  for  regulated  firms  to  comply  and  administrative 
methods  for  resolving  disputes  (17:450).  The  system  in 
which  an  agency  operates  affects  its  performance  and  what  it 
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TABLE  3.1 


Guidelines  for  Establishing  Performance  Measures 


1.  Seek  substitutes  for  "profit"  which  can  be  measured  and 
analyzed  (34:10).  For  a  public  agency  this  might  be 
equivalent  to  net  income  or  operating  surplus  (31:175). 

2.  Initial  attempts  at  developing  organizational  measures 
might  include  project  completion  and  timetables  (31). 

3.  Consider  using  "result"  measures  as  opposed  to  "process" 
measures.  Results  measures  are  better  because  they  are 
readily  observable,  reflect  the  quality  of  work,  are 
congruent  with  organizational  objectives,  and  emphasize  end 
results  (3:520;  31:163). 

4.  Ensure  there  is  a  variety  of  measures  (10:273;  25:385). 
Agency  performance,  evaluated  using  only  a  few  mechanisms, 
may  actually  be  declining  while  indicators  show  a  positive 
trend  (41:73) . 

5.  Attempt  to  make  measures  applicable  to  organizations 
with  similar  operational  characteristics  (29:143). 

6.  Performance  measures  should  be  established  using  input 
of  personnel  at  the  decision  making  unit  level  (31:160). 

7.  Measures  should  be  limited  to  those  for  which  reliable 
information  is  available  and  management  has  the  ability  to 
control  (4:205;  25:385). 

8.  Keep  a  balance  between  effectiveness  and  efficiency 
measures  when  possible  (29:142).  Effectiveness  measures 
should  be  "cost  free"  (25:381). 

9.  Candidate  measures  should  meet  certain  criteria  when 
using  a  quantitative  model  such  as  DEA  and  CFA  (6). 

a.  Outputs  represent  important  DMU  goals. 

b.  Measures  apply  to  DMUs  under  consideration  and 
exist  in  non-zero  amounts. 

c.  Inputs  represent  resources  used  by  the  DMUs 
toward  attaining  outputs. 

d.  Changes  in  inputs  will  result  in  corresponding 

changes  in  outputs. 

e.  The  magnitude  of  pnysical  input  and  output 
quantities  is  represented. 

f.  The  input  and  output  quality  are  represented. 


can  use  foe  measurement.  In  the  broadest  sense,  agency 
performance  is  judged  by  how  well  public  interests  are 
protected.  The  regulatory  agency  is  a  collection  of 
Decision  Making  Units  (DMUs)  which  are  responsible  for 
various  clients,  some  of  them  with  divergent  viewpoints. 

One  agency  goal  is  to  satisfy  clients  without  antagonizing 
competitive  interests  (38:58).  See  Figure  3.1  for  an 
overview  of  the  system  in  which  the  EPA  operates. 

There  are  3  main  clients  of  the  EPA  (see  Figure  3.1) 
which  have  an  influence  on  how  the  agency  is  evaluated. 

1.  Public  -  The  public  is  the  ultimate  determinant  of 
the  regulatory  approach,  yet  it  may  lose  interest  when 
dealing  with  technical  matters.  As  the  public's  affluence 
increases,  they  expect  more  of  government  to  improve  the 
living  standard  (41:11). 

2.  Interest  Groups  -  The  public  commonly  provides 
input  through  and  support  to  "institutionalized"  interest 
groups  which  can  deal  with  technical  issues.  The  groups 
have  become  more  sophisticated  and  changed  their  focus  to 
specific,  subtle  forms  of  pollution  (41:12). 

3.  Regulated  Community  -  Most  regulated  organizations 
prefer  to  have  comprehensive,  well-enforced  regulations  so 
less  conscientious  organizations  are  not  given  advantages 
(17:451).  Regulated  entities  evaluate  the  EPA  in  this 
regard  and  whether  pollution  control  hurts  economically. 
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Figure  3.1:  EPA  Regulatory  System 
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Similarly,  the  EPF  has  the  responsibility  of  ensuring 
that  base  organizations  comply  with  EPA  and  Air  Force 
environmental  regulations.  Various  methods  of  enforcement 
are  available,  including  working  through  committees, 
threatening  potential  EPA  involvement,  or  elevating  to  the 
base  commander.  The  EPF  is  in  the  tenuous  position  of  not 
only  assisting  and  advising,  but  attempting  to  regulate 
other  organizations. 

The  EPF  has  clients  similar  to  the  EPA's,  such  as  a 
regulated  community  (e.g.,  maintenance)  and  allies  (e.g., 
bioenvironmental  engineering).  The  EPF  has  to  deal  with  the 
public  and  the  base  population,  although  their  interest  is 
usually  only  piqued  when  something  goes  wrong.  Recently 
interest  groups  have  shown  more  interest  in  DOD  as  base 
environmental  situations  are  increasingly  publicized.  The 
EPFs  are  often  confronted  by  public  and  interest  group 
coalitions.  See  Figure  3.2  for  an  overview  of  the  EPF 
system  which  is  analogous  to  EPA's. 

Chapter  II  listed  some  performance  measurement 
problems  of  public  agencies.  The  EPA  and  other  regulatory 
agencies  are  confronted  with  additional  conditions  which 
affect  their  ability  to  perform  as  expected  and  to  measure 
this  performance.  Analogous  conditions  exist  for  other 
environmental  organizations  such  as  the  EPF. 

1.  Government  Emphasis  -  Various  federal  admini¬ 
strations  place  different  emphasis  on  regulation  and 


environmental  matters,  so  the  EPA  may  need  other  enforcement 
means,  such  as  economic  incentives,  to  ensure  effectiveness 
(17).  Reduced  government  emphasis  on  regulation  may  make 
the  EPF  job  easier,  unless  the  deemphasis  results  in  less 
EPF  funds. 

2.  Form  of  Output  •  Regulatory  agencies  predominately 
provide  services  which  are  not  readily  measured.  Some  other 
public  agencies  are  more  product  oriented,  such  as  the 
Tennessee  Valley  Authority  which  produces  electricity 
(36:247).  The  EPF  also  provides  a  service,  not  a  product  as 
a  maintenance  organization  does. 

3.  Change  in  Agency  Role  *  the  EPA's  performance  is 
difficult  to  track  considering  the  frequent  changes  in 
mission  goals  and  enforcement  methods.  For  example,  the 
Agency  has  vacillated  between  an  adversarial  and  ally  role 
in  its  relation  with  business  (33:165).  The  EPF's  role 
changes  with  changes  in  its  parent  agency  and  in  the  EPA's 
regulatory  approach. 

4.  Difference  in  Agency  Programs  •  the  EPA's  programs 
vary  in  their  suitability  for  measurement.  The  enforcement 
program  is  relatively  "soft"  compared  to  the  grant  program 
which  provides  funds  for  building  wastewater  treatment 
plants  (24:126).  The  EPF  experiences  similar  problems  in 
comparable  base  programs. 

5.  Politics  -  Regulatory  agencies  are  in  political 
environments  where  vagueness  is  often  preferable  to  clarity. 


Publicly  defining  specific  organizational  performance  may 
offend  someone  internal  or  external  to  the  agency  (1:6; 
31:161).  The  EPF  must  deal  with  politics  on  and  off  base  in 
the  volatile  environmental  area. 

Developing  measures  is  not  an  easy  task  for  environ¬ 
mental  organizations  considering  all  these  performance 
related  difficulties.  "Trial  and  error"  has  been  the 
development  technique  for  many  organizations.  One  public 
utility's  approach  to  being  publicly  responsible  was  to 
develop  "supplemental  measures  of  performance."  For 
environmental  quality,  they  measured  actions  taken  against 
the  firm,  including  number  of  lawsuits  and  amount  of  fines 
(31:154).  Lawsuits  were  soon  dropped  because  the  utility 
had  little  control  over  someone  wanting  to  sue. 

The  EPA  has  found  it  desirable  to  have  a  variety  of 
enforcement  options  that  range  from  gentle  incentives  to 
severe  penalties  for  criminal  violations.  (33:132).  In  the 
1960s,  the  government  relied  on  legal  enforcement  of 
stringent  pollution  standards.  By  the  1970s,  the  poor 
success  of  the  earlier  approach  caused  support  to  switch  to 
other  enforcement  methods  (36:231).  Shifting  focus  to 
economic  incentives,  such  as  effluent  charges,  reduced  the 
number  of  enforcement  cases.  (24:130).  The  EPA  was  able  to 
increase  efficiency  while  still  effectively  reducing 
pollution.  Thus,  numbers  of  violations  issued  may  not 
always  be  a  good  performance  indicator  (by  itself)  for  EPA. 


In  1982,  the  EPA  shifted  to  an  enforcement  program 
which  relies  on  "managing  for  environmental  results" 
(19:605).  The  quantitative  methods  were  balanced  with 
reports  of  how  effectively  managers’  actions  serve  to 
protect  the  environment.  The  three  main  aspects  of  the 
program  are  an  accountability  system,  environmental 
management  reports,  and  indicators  of  environmental  results 
(19:605).  The  accountability  system  and  the  management 
reports  are  similar  to  performance  evaluations  discussed 
earlier . 

Of  primary  interest  are  the  indicators  of  environ¬ 
mental  results  which  represent  an  attempt  by  the  EPA  to 
improve  objective  measurements.  "Programmatic  indicators" 
are  paperwork  actions  taken  by  the  Agency,  such  as  permit 
revisions  or  violation  issuances.  "Ultimate  impact 
indicators"  include  number  of  pollution  incidents  or  amount 
of  air  pollution  reduction  obtained.  While  the  ultimate 
indicators  are  not  as  easily  obtained,  the  EPA's  attempts  to 
use  the  best  combination  of  data  available  to  see  if 
environmental  objectives  are  being  met  (19:605).  However, 
results  measures  are  often  difficult  to  evaluate  when 
considering  the  impact  of  the  program,  not  just  the  output 
(3:564).  For  example,  reducing  emissions  from  smoke  stacks 
is  an  output  relatively  easy  to  measure,  while  long  term 
impacts  of  reduced  air  pollution  are  difficult  to  measure. 


As  part  of  this  new  regulatory  approach,  the  EPA 
developed  an  "environmental  auditing"  program.  The  program, 
which  many  private  companies  and  public  agencies  already 
use,  takes  a  snapshot  of  compliance  status  at  a  given  time. 
Audits  are  considered  a  more  efficient  means  of  ensuring 
compliance  because  infractions  are  caught  early  before 
serious  consequences  occur  (18:509).  While  audits  could  be 
characterized  as  performance  evaluations,  they  contain 
measures  such  as  out-of-compliance  pollution  control 
equipment  which  can  be  quantitatively  analyzed.  Ultimately 
the  success  of  an  environmental  function  is  measured  by  its 
performance  during  audits.  Developing  measures  and 
periodically  evaluating  them  prepares  an  EPF  for  such 
audits. 

The  above  discussion  provided  some  general  guidelines 
for  establishing  productivity  measures  to  be  used  by  public 
agencies.  Some  specific  output  measure  candidates  for  the 
EPA  or  other  environmental  agencies  are  listed  at  Table  3.2. 
(Input  measures  are  not  considered  because  they  are  fairly 
standard  among  organizations.)  Some  of  the  output 
candidates,  which  could  be  modified  for  EPF  use,  are 
considered  in  Chapter  V. 

Conclusion 

Ironically,  much  of  the  literature  appears  to  be  on 
how  to  increase  productivity  with  less  emphasis  on  measuring 


TABLE  3.2 


Possible  Environmental  Agency  Output  Measures 


1.  Funding  level,  percent  increase  or  percent  of  requested 
amount  ( 31 ) . 

Organization  cannot  function  without  proper  funding.  Shows 
how  well  EPA  lobbies  Congress  and  constituents  for  support. 

2.  Monitor  and  improve  environmental  quality  (26:169). 

One  of  EPA's  responsibilities  under  federal  law.  Difficult 
to  measure  as  stated. 

3.  Number  of  monitoring  reports  or  compliance  measurements. 
Alternative  to  above  which  is  more  easily  measured. 

4.  Fines  and  penalties  assessed  (26:170). 

If  firms  do  not  comply,  then  this  measure  reflects 
EPA's  ability  to  assess  penalties.  Ultimate  goal  of 
measuring  environmental  quality  might  not  be  reflected. 

5.  Number  of  Violations  Issued  (33). 

Necessary  when  enforcing  compliance  with  environmental 
standards.  Excessive  number  (too  many  violators)  may 
indicate  something  is  wrong  with  the  approach. 

6.  Dollar  loss  from  environmental  degradation  24 ;  35). 

Indicates  trend  in  overall  protection  program,  it  is 
difficult  to  measure  or  place  dollar  figure  on  loss. 

7.  Project/program  completion  times  (31:175). 

A  rather  fundamental  concept  of  attempting  to  complete 
projects  on  time.  It  is  appropriate  for  new  programs  or 
other  situations  where  measures  are  not  well  defined. 

8.  Amount  of  inter/intra-agency  coordination  required 
(31:176) . 

Measures  amount  of  communication  between  or  in  agencies 
which  prevents  program  overlap.  Relates  to  efficiency 
of  providing  government  services,  but  appears  difficult  to 
measure. 


TABLE  3.2  (Continued) 


9.  Input  effectiveness  (38:57). 

Number  of  internal  messages  needed  to  solve  a  problem. 

10.  Processing  capacity  (38:57). 

Average  waiting  time  per  message. 

11.  Agency  responsiveness  (38:57). 

Client  satisfaction  index  combining  waiting  time  and  agency 
response. 

12.  Numoer  of  case  closings  (10) 

An  example  for  the  EPA  would  be  the  number  of  compliance 
investigations  completed. 


wnat  is  to  be  improved.  Many  articles  disregard  what 
measures  are  to  t>e  evaluated  and  now  performance  measurement 
will  be  accomplisned.  Obviously  these  are  key  aspects  wnen 
looking  at  sucn  management  functions  as  scneduling  ana 
communication.  Improvement  can  only  occur  when  there  is  an 
understanding  of  wnat  aspects  are  deficient  ana  wnen  to 
concentrate  work  on  a  specific  area. 

The  nature  of  regulatory  agencies  makes  performance 
measurement  difficult.  Regardless  of  the  problems,  it  is 
important  tnat  agencies  have  some  type  of  evaluation  program 
and  attempt  quantitative  measurement.  Measuring  means 
emphasizing  program  goals  ana  priorities  by  attracting 
attention  to  what  management  considers  important.  "Any 
measure  is  imperfect.  As  long  as  tne  end  result — tne  goal — 
is  clear,  however,  almost  any  measure  is  acceptable." 
(31:174) . 
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IV.  Methodology 


Overview 

As  discussed  in  Chapters  II  and  III,  productivity 
measurement  in  public  organizations  is  a  complex  problem. 
Multiple-inputs,  multiple-outputs  and  few  tangible  outputs 
make  evaluations  of  service-oriented  Air  Force  Civil 
Engineering  organizations  difficult.  Further  complications 
arise  when  assessing  the  Environmental  Planning  Function 
(EPF)  because  of  its  broad  charter  (e.g.,  protect  the 
environment) . 

In  approaching  the  problem  of  evaluating  EPF 
productivity,  these  investigative  questions  need  to  be 
answered : 

a.  Of  the  measurement  models  available  (such  as 
regression,  ratio  analysis,  Data  Envelopment  Analysis,  and 
Constrained  Facet  Analysis)  which  provides  the  best  method 
of  analyzing  EPF  performance? 

b.  what  constitutes  a  set  of  inputs  and  outputs  to 
form  a  complete  representation  of  Air  Force  EPF  performance? 

After  discussing  the  analytical  models,  the  chapter 
will  provide  a  methodology  for  specifying  and  identifying 
candidate  EPF  input  and  output  measures.  A  data  collection 
plan  and  proposed  procedures  for  accomplishing  the  data 
analysis  are  outlined. 


41 


Selection  of  Analysis  Method 

Few  analytical  models  compare  multiple  inputs  to 
multiple  outputs  while  accounting  for  the  interaction  of 
these  variables.  Data  Envelopment  Analysis  (DEA)  developed 
by  Charnes,  Cooper,  and  Rhodes  (12)  accomplishes  this 
complex  analysis,  making  the  use  of  multiple  variables  more 
meaningful.  Bessent,  Bessent,  Clark  and  Elam  (7)  refined 
DEA  into  the  more  advanced  CFA  model.  DEA  and  CFA  have  been 
rated  superior  to  regression  and  ratio  analysis,  exemplified 
by  their  success  in  evaluating  school  systems  and  Air  Force 
organizations  as  noted  in  Chapter  II  (5;  11;  28).  The  next 
three  sections  discuss  the  operation  and  attributes  of  DEA 
and  CFA  and  explain  why  they  were  selected  as  the  appro¬ 
priate  model  for  evaluating  EPFs.  A  brief  comparison  of 
various  analytical  methods  is  found  in  Appendix  A  (14),  and 
several  theses  (11;  23;  22)  provide  more  details  on  these 
methods . 

Data  Envelopment  Analysis  Operation  (15) 

DEA  was  developed  to  evaluate  the  relative  efficiency 
of  public  sector  organizations  performing  similar  type 
functions.  Using  a  computerized  linear  programming  model, 
DEA  can  evaluate  multiple  inputs  and  outputs  and  their 
interrelationships  simultaneously.  The  model  generates  an 
efficiency  rating  for  each  organization,  designated  a 
Decision  Making  Unit  (DMU),  compared  to  the  other 


organizations  (DMUs).  DEA  compares  all  DMUs  to  locate  the 
best  performing  units,  those  which  produce  the  greatest 
output  from  the  least  input.  Then  the  model  compares  the 
remaining  units  to  the  best  units  and  provides  a  single 
aggregate  measure  of  relative  efficiency  for  each  DMU.  The 
highest  possible  rating  a  DMU  can  achieve  is  1.0  in 
comparison  to  the  other  most  efficient  units.  Since  the 
rating  is  an  input/output  ratio,  the  less  efficient 
organizations  will  have  a  rating  less  than  1.0  and  greater 
than  0.0  ( 15 )  . 

Figure  4.1  is  a  simplified  two*dimensional  comparison 
of  DMU  efficiency  using  DEA.  Each  unit  has  two  inputs  (XI 
and  X2 )  and  one  output  (Y).  DEA  reduces  the  DMU  inputs  by 
the  amount  of  output  each  produces  so  that  all  output  values 
are  one  (1.0).  Each  DMU  is  plotted  according  to  its 
input/output  ratio- -Xl/Y  on  the  abscissa  and  X2/Y  on  the 
ordinate.  The  more  efficient  units  are  closer  to  the  origin 
since  they  use  less  input  (XI  or  X2)  for  a  given  output  (Y) . 
A,  B  and  C  are  given  ratings  of  1.0,  while  D  is  rated  less 
than  1.0  in  comparison  to  A,  B  and  C.  This  comparison  is 
accomplished  by  drawing  a  line  between  the  most  efficient 
units  (i.e.,  a  frontier)  and  comparing  D  to  the  line.  D's 
rating  is  the  line  sector  OQ  value  over  the  line  sector  OD 
value.  In  this  case,  the  ratio  is  approximately  3.5  (OQ) 


divided  by  5.0  (OD)  for  an  efficiency  rating  of  0.70. 
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Figure  4.1:  Relative  Efficiency  of  One  Output, 
Two  Input  Organizations 
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Other  benefits  of  DEA  include  its  scaling  and 
weighting  characteristics.  The  model  is  able  to  compare 
multiple  inpu-ts  and  outputs  of  a  DMU  without  having  them  in 
a  common  scale  or  unit  of  measurement.  This  is  accomplished 
by  comparing  the  same  measures  of  one  DMU  to  the  same 
measures  of  other  DMUs.  As  pointed  out  in  Chapter  III,  non* 
profit  DMUs  cannot  use  solely  monetary  comparisons,  so  the 
scaling  characteristic  is  important.  Also,  the  model  does 
not  require  a  prior  weighting  of  the  measures  according  to 
their  importance.  Instead,  the  model  derives  from  the  data 
what  weights  should  be  assigned  to  each  measure.  These 
weights  can  be  used  to  identify  DMU  inefficiency  and  DMU 
measures  which  can  be  changed  to  increase  efficiency  (7). 


Constrained  Facet  Analysis  Operation 

The  CFA  model  is  an  extension  of  the  DEA  model, 
designed  to  eliminate  the  DEA  limitations.  DEA’s  primary 
limitation  is  the  inability  to  fully  explain  (i.e., 
envelope)  all  DMUs,  giving  some  units  inflated  efficiency 
ratings  (7).  DEA  can  rate  a  DMU's  efficiency  in  comparison 
to  a  "neighborhood"  of  the  most  efficient  units.  However, 
any  unit  which  does  not  have  a  comparable  mix  of  inputs  and 
outputs  to  the  neighborhood  will  not  be  evaluated.  DMU  E  in 
Figure  4.1  is  an  example  of  a  DMU  not  enveloped  by  DEA. 

Since  the  ray  from  the  origin  to  DMU  E  does  not  cross  the 
relative  efficiency  frontier,  DEA  cannot  determine  its 
relative  efficiency  (14). 
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CFA  goes  beyond  the  initial  step  of  identifying  the 
efficiency  frontier  and  the  relative  DMU  efficiencies.  CFA 
goes  through  an  iterative  process  of  comparing  less 
efficient  DMUs  with  those  along  the  efficiency  frontier. 

For  those  units  which  do  not  have  a  facet  of  the  frontier  to 
be  compared  against,  CFA  "extends"  the  facet  so  that  the 
unit  can  be  compared  against  units  with  similar  mixes  of 
measures.  Facet  A-B-C  has  been  extended  beyond  C  by  CFA  in 
Figure  4.1  to  envelop  DMU  E  and  compare  it  to  other  DMUs. 

DMU  E  is  compared  to  point  S  for  an  upper  bound  value  and  to 
point  R  for  a  lower  bound  value.  Thus,  CFA  attempts  to  rate 
a  DMU  with  estimates  of  efficiency  even  if  a  DMU  has  an 
input  or  output  mix  which  causes  it  to  be  unbounded  by  the 
relative  efficiency  frontier.  The  value  of  the  information 
to  managers  is  enhanced  since  inefficient  DMUs  of  various 
input/output  mixes  can  be  compared  to  at  least  somewhat 
comparable  units  (14).  Because  of  CFA's  capability  to  make 
more  organizational  comparisons,  this  research  will  attempt 
to  use  primarily  CFA  for  data  analysis. 


DEA  and  CFA  Attributes 


The  following  is  a  summary  of  DEA/CFA  attributes  which 
make  the  models  desirable  for  comparing  public  organizations 
(14;  22:24-25): 

1.  The  models  compare  multiple  inputs  and  outputs 
simultaneously  providing  single  aggregate  ratings  (ratios). 
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2.  Efficiency  frontiers  are  based  on  empirical  data 
rather  than  on  preassigned  ideals. 

3.  "Neighborhood"  comparisons  ensure  that  DMUs  are 
mainly  compared  to  units  with  similar  measures. 

4.  The  models  are  less  biased  because  weights 
reflecting  value  judgments  are  not  assigned  a  priori. 

5.  The  models  indicate  output  shortages  and  input 
overages  so  that  corrective  actions  for  inefficient  DMUs  can 
be  taken. 

6.  The  models  handle  non-commensur ate  measures  which 
are  frequently  found  in  non-profit  organizations. 

7.  The  inefficient  units  are  assigned  the  highest 
possible  efficiency  rating,  giving  them  the  benefit  of  doubt 
concerning  measurement  error. 

8.  CFA  attempts  to  rate  all  DMUs  by  providing  an 
approximate  comparison  frontier  for  unbounded  DMUs. 

Input  and  Output  Measures  Selection  Process 

A  critical  step  in  analyzing  productivity  is  the 
specification  of  candidates  as  input  and  output  measures. 
This  research  attempted  to  find  quantifiable  measures, 
easily  accessible  to  managers,  that  can  be  used  to  assess 
their  units'  productivity.  Development  of  EPF  measures 
followed  a  formalized  identification  and  selection 
methodology  developed  by  Bessent,  Bessent,  and  Clark  (6). 

The  remainder  of  this  section  follows  the  methodology 
flowchart  at  Figure  4.2. 
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Reprinted  from  Specification  of  Inputs  and  Outputs 

in  Data  Envelopment  Analysis  (6) 


Figure  4.2:  Specification  of  Inputs  and  Outputs 


The  initial  row  in  the  flowchart,  describing  current 
information  available,  is  important  to  thoroughly  study 
because  of  the  nebulous  nature  of  the  DMU  under  investiga¬ 
tion.  "Available  Measures,"  "Studies  and  Reports,"  and 
"Formal  Statements  of  DMLJ  Goals  and  Services"  form  the  basis 
of  understanding  the  EPF  operations  and  lead  to  a  descrip¬ 
tion  of  the  managerial  context  (next  block  in  the  flow¬ 
chart).  The  managerial  context  is  that  aspect  of  the  DMU's 
overall  function  which  is  to  be  evaluated. 

The  next  step  is  to  generate  a  comprehensive  set  of 
input  and  output  measures  based  on  the  managerial  context. 
Providing  a  comprehensive  list  of  potential  measures  is 
important  for  success  in  modeling  since  omission  of  a 
critical  input  or  output  could  distort  results.  Screening 
of  measures  for  duplications  or  inappropriateness  occurs  in 
the  next  step. 

According  to  the  flowchart,  the  measures  should  be 
tested  after  an  acceptable  set  has  been  established. 
Unacceptable  results  indicate  a  need  to  find  more  measures 
or  modify  existing  measures  (assuming  all  steps  were  correct 
to  that  point).  In  essence,  this  research  is  providing 
these  testing  and  analysis  steps  and,  if  successful,  will 
result  in  a  set  of  measures  for  future  use  by  EPF  managers. 
Chapter  V  actually  steps  through  the  flowchart  to  arrive  at 
an  acceptable  list  of  EPF  measures. 


Data  Population 


The  population  for  this  research  is  the  CONUS  Air 
Force  installations  under  eight  Major  Commands.  Wide 
variability  exists  among  installation  missions  and 
environmental  protection  programs.  AFLC's  depot-level 
maintenance  activities  overshadow  most  smaller  bases  in 
environmental  significance.  Regardless  of  the  variability, 
this  set  of  installations  was  selected  for  several  reasons. 
First,  it  was  important  to  examine  these  measures  to 
determine  if  they  apply  to  all  bases  regardless  of  size. 

All  Air  Force  installations  have  the  same  objective  of 
environmental  protection,  and  consequently  should  have 
similar  measures.  Secondly,  Headquarters  Air  Force  receives 
environmental  status  reports  from  all  CONUS  installations. 
Parts  of  these  reports  form  the  basis  for  some  of  the 
measures.  Validating  these  measures  means  Air  Force  can  use 
the  status  reports  and  the  corresponding  measures  for 
comparisons  of  all  installations. 

The  third  reason  has  to  do  with  the  number  of 
observations  needed  for  DMU  efficiency  analysis.  The  CFA 
model  requires  a  minimum  number  of  observations  to  establish 
an  efficiency  frontier.  The  rule  of  thumb  is  that  observa¬ 
tions  should  be  twice  the  number  of  input  and  output 
measures.  If  all  the  installations  reply,  a  total  of  85 
observations  will  be  available  for  analysis,  allowing  42 
inputs  and  output  measures  to  be  used  in  the  CFA  analysis. 


Even  though  all  the  measures  may  not  be  used  as  variables 
for  each  analysis  run,  the  more  measures  used  the  better  the 
characterization  of  the  DMU.  Therefore,  using  most  of  the 
CONUS  installations  will  allow  the  use  of  a  greater  number 
of  measures  to  describe  EPFs. 

The  selection  of  the  EPF  as  the  DMU  was  done  to 
confine  the  analysis  to  a  DMU  with  a  set  of  related 
objectives.  The  other  functions  within  the  Environmental 
and  Contract  Planning  Section  have  responsibilities 
dissimilar  to  the  environmental  and  resource  protection 
responsibilities  of  the  EPF  (see  Figure  4.3). 


The  contract  planning  function  has  few,  if  any,  of  the  same 
objectives  as  the  EPF.  The  comprehensive  planning  function 
has  many  responsibilities,  such  as  Air  Installation 
Compatible  Use  Zone,  which  would  require  more  measures  to 
analyze.  The  key  point  is  that  the  jobs  within 
Environmental  and  Contract  planning  are  so  diverse  that  too 
many  variables  (measures)  would  be  required  to  adequately 
describe  the  Section  and  still  use  the  CFA  model 
effectively. 

Use  of  the  PEA  and  CFA  Models 

DEA  and  CFA  were  selected  as  analysis  models  so  that 
multiple  inputs  and  outputs  needed  to  describe  EPFs  can  be 
analyzed.  The  models  will  identify  the  inefficient  DMUs  and 
inputs  and  outputs  which  contribute  to  the  inefficiency. 

The  models  take  into  account  the  interactions  between 
variables.  The  significance  of  each  input  and  output 
variable  is  determined  by  the  models  and  will  be  unique  for 
each  organization.  Weights  do  not  need  to  be  assigned  to 
the  variables  since  the  weights  are  determined  by  the  models 
when  they  assign  multiplier  values  to  the  inputs  and 
outputs. 

Using  surveys,  data  will  be  solicited  from  85 
installations  regarding  the  measures  chosen  to  be  analyzed. 
The  data  will  be  analyzed  using  the  computer  program, 
Productivity  Assessment  Support  System,  which  is  an 


"integrated  system  for  performing  productivity  analyses  [DEA 
and  CFA]  for  units  that  utilize  the  same  kinds  of  inputs  to 
produce  the  same  kinds  of  output  over  the  same  period  of 
time"  (8).  Productivity  Assessment  Support  System  is 
written  in  dBase  III  computer  language  and  operates  on  an 
IBM-AT.  Since  the  system  is  restricted  to  20  variables,  not 
all  measures  can  be  compared  during  each  run.  Depending  on 
the  analysis  desired,  either  some  of  the  compatible  measures 
will  be  combined  before  a  run  or  only  selected  inputs  will 
be  compared  against  selected  outputs  during  a  specific  run. 

The  analysis  results  must  be  interpreted  for  applica¬ 
tion  to  the  function,  since  the  models  cannot  explain  what 
caused  the  measure's  inefficiency  or  how  to  correct  it. 

After  analyzing  the  data,  the  research  will  evaluate  whether 
the  models  can  be  used  to  provide  information  to  managers 
for  improving  their  EPFs.  Inefficient  installation  units 
may  be  able  to  change  the  input  surpluses  or  output  short¬ 
ages  for  measures  identified  by  the  models.  Headquarters 
Air  Force  or  Major  Commands  may  be  interested  in  what 
general  performance  problems  are  confronting  the 
installations. 

Summary 

This  chapter  answered  investigative  question  2  by 
evaluating  analysis  methods  and  selecting  the  DEA  and  CFA 
models  as  the  desired  technique.  It  further  discussed  how 


DEA  and  CFA  would  be  used  for  analyzing  EPFs  from  CONUS 
installations  which  were  selected  as  the  data  population. 

As  an  initial  step  in  answering  investigative  question  3, 
the  discussion  covered  the  methodology  to  be  used  for 
selecting  DMU  measures.  Chapter  V  discusses  how  the  input 
and  output  measures  were  selected,  then  narrowed  to  the 
number  and  type  which  best  represent  EPFs  and  best  suit  the 
models'  capabilities. 


V.  Selection  of  Input  and  Output  Measures 
and  Data  Collection 

Introduction 

This  chapter  describes  the  process  for  selecting  the 
candidate  EPF  measures  and  refining  that  list  to  the 
measures  which  most  accurately  portray  EPF  operations.  The 
selection  process  will  follow  the  procedure  (6)  described  in 
Chapter  IV.  The  refinement  process  involved  the  expertise 
of  Air  Force  personnel  familiar  with  EPFs.  Developing 
measures  was  an  iterative  process  requiring  numerous 
revisions,  since  few  efforts  have  been  made  to  characterize 
the  EPF  in  objective  terras  such  as  inputs  and  outputs.  The 
chapter  finishes  by  describing  what  data  were  needed  for  the 
selected  variables  and  where  the  information  was  obtained. 
This  fulfills  investigative  question  3. 

Information  Available  for  Measurement  Development 

Reviewing  available  measures,  DMU  goals  and  services, 
and  DMU  studies  and  reports  helps  to  describe  the  managerial 
context  leading  to  the  generation  of  a  list  of  measures  (see 
the  flowchart  at  Figure  4.2).  Chapter  III  addressed  the 
first  item  by  reviewing  measures  which  could  reasonably  be 
applied  to  the  EPF.  This  section  describes  EPF  goals  and 
services,  EPF  reports,  and  the  EPF  managerial  context. 

As  an  implementing  document,  Air  Force  Regulation  19- 
1,  Pollution  Abatement  and  Environmental  Quality, 


established  the  formal  environmental  protection  program. 

AFR  19-1  "states  and  assigns  responsibilities  for  the 
development  of  an  organized,  integrated,  multidisciplinary, 
environmental  protection  program  to  make  sure  that  the  Air 
Force  conduct  its  activities  in  a  manner  that  protects  and 
enhances  environmental  quality"  (39:1).  The  regulation 
emphasizes  that  Air  Force  policy  is  to  comply  with  not  only 
the  letter  but  the  spirit  of  federal,  state  and  local 
environmental  laws  and  regulations.  Further,  Air  Force 
installations  are  expected  to  demonstrate  leadership  in 
correcting  environmental  ills,  controlling  pollution,  and 
more  importantly,  reducing  actual  or  potential  causes  of 
pollution  (39:2).  Thus,  the  overall  goal  of  environmental 
protection  is  not  to  increase  production  of  a  product,  but 
to  increase  service  for  the  agency  to  be  in  compliance  with 
environmental  regulations.  To  meet  the  objectives  of  this 
goal,  the  Air  Force  is  avoiding  pollution  of  the  air,  land, 
and  water.  In  a  sense,  the  objectives  are  comparable  to 
cost  avoidance  objectives. 

Current  means  of  documenting  Air  Force  environmental 
performance  are  limited  primarily  to  two  formal  reports:  1) 
Pollution  Status  Report,  RCS:  DD-I(SA)  1381,  and  2)  Defense 
Environmental  Status  Report,  RCS:  DD-M(A)  1485.  The  first 
report  provides  the  status  of  pollution  abatement  projects. 
The  base  projects  listed  in  the  report  are  heavily  dependent 


on  the  age  of  base  facilities  and  on  the  base  mission; 
therefore,  the  report  is  a  poor  indicator  of  overall  EPF 
functioning . 

The  Defense  Environmental  Status  Report  is  structured 
to  show  how  well  the  environmental  program  is  meeting  the 
intent  of  AFR  19-1  and  other  directives.  The  report  is 
divided  into  six  environmental  programs  with  goals  assigned 
to  each  of  the  following  areas; 

1.  Clean  Air 

2.  Clean  Water 

3.  Solid  and  Hazardous  Waste 

4.  Installation  Restoration 

5.  Safe  Drinking  Water 

6.  Environmental  Auditing  Management 

Appendix  B  is  a  complete  listing  of  areas  and  goals.  For 
the  most  part  these  goals  represent  environmental  compliance 
objectives  and  methods  including  operational  requirements 
(such  as  correcting  pollution  sources)  and  administrative 
requirements  (such  as  applying  for  permits).  Part  2  of  the 
Defense  Environmental  Status  Report  addresses  management 
indicators  for  the  various  goals  in  Part  1.  Most  of  the 
management  indicators  are  comparisons  of  compliance  from 
period  to  period,  an  indication  of  the  importance  that  the 
Air  Force  assigns  to  avoiding  environmental  problems. 

Another  performance  evaluation  technique  being 
developed  by  the  Air  Force  is  the  Environmental  Compliance 
and  Management  Program.  The  crux  of  the  program  is  auditing 
base  programs  to  determine  their  compliance  with 


environmental  regulations.  The  program  is  referenced  as 
part  of  the  sixth  Defense  Environmental  Status  Report  goal 
and  is  similar  to  the  Environmental  Protection  Agency's 
audit  program  discussed  in  Chapter  III.  The  present 
evaluation  method,  as  depicted  in  Table  5.1,  only  provides  a 
binary  (i.e.,  in  or  out  of  compliance)  analysis  which  is  not 
well  suited  for  DEA/CFA.  Also,  the  Environmental  Compliance 
and  Management  Program  is  presently  only  in  a  prototype 
stage  and  comprehensive  information  would  not  be  available. 
However,  the  emphasis  by  Air  Force  on  implementing  the 
program  indicates  the  importance  of  using  compliance  as  a 
success  criterion.  Further  development  of  the  Environmental 
Compliance  and  Management  Program  could  eventually  result  in 
its  use  as  either  a  measure  or  a  data  source  for  DEA/CFA. 

Environmental  protection  performance  has  been 
evaluated  in  a  general  sense  for  many  years  as  the  ability 
to  stay  in  compliance  (i.e.,  to  avoid  violations).  The 
senu-annual  environmental  reports  provided  this  information 
to  headquarters  levels.  But  the  base  level  managers  have 
not  had  a  method  to  evaluate  their  efficiency  in  meeting 
these  standards,  especially  in  comparison  to  other  DMUs. 
Therefore,  the  managerial  context  defined  for  the  efficiency 
model  is  the  measure  of  resource  utilization  in  attempting 
to  attain  total  compliance. 


COMPLIANCE  SUMMARY  TABLE 


Substantially  Out  of 

in  Compliance*  Compliance 


Air  Emissions 

fuel  burners  X 

incinerators 

VOC 

other 

Wastewater  Discharges 


sanitary  sewage  X 

stormwater 

industrial  wastewater 
Solid  Waste  X 

Hazardous  Waste 

accumulation  points  X 

TSD  facilities  X 

PCB  X 

Pesticide  X 

POL  Management 

SPCC  X 

leaking  tanks/pipes  X 

Drinking  Water  X 

Hazardous  Materials  Management  X 


In  compliance  with  regulatory  requirements  where  a  few 
minor  (technical  or  administrative)  exceptions  exist. 


XX  XX 


The  next  step  in  the  flowchart,  developing  a 
comprehensive  list  of  input  and  output  candidates,  was 
assisted  by  the  participants  of  the  functional  review 
workshop  for  the  Environmental  and  Contract  Planning 
Section,  10-14  March  1986.  The  workshop  was  one  of  a  series 
held  at  Headquarters  Air  Force  Engineering  and  Services 
Center,  Tyndall  Air  Force  Base,  under  the  auspices  of 
Project  IMAGE.  The  participants  identified  an  exhaustive 
list  of  organizational  tasks,  problems,  and  potential 
corrective  measures  (Appendix' C) .  Based  on  the  results  of 
this  workshop  and  the  author's  experience  in  the  environ¬ 
mental  field,  a  comprehensive  list  of  inputs  and  outputs  was 
generated  (see  Appendix  D) .  Key  ingredients  in  the  develop¬ 
ment  were  the  analysis  of  measures  in  Chapter  III  and  the 
information  sources,  such  as  the  status  reports,  mentioned 
earlier  in  this  Chapter. 

Output  measures  can  be  classified  primarily  as  either 
results  measures  or  process  measures.  Results  measures 
reflect  DMUs  organizational  objectives,  while  process 
measures  reflect  organizational  activities.  Because  the 
production  function  for  the  EPF  is  difficult  to  identify, 
measure,  and  quantify,  process  measures  were  not  considered 
satisfactory  by  themselves.  Many  of  the  outputs  were 
devised  as  results  measures  so  they  would  indicate 
compliance  status.  A  combination  of  process  and  results 
measures  helps  to  thoroughly  describe  a  DMU's  activities. 


although  results  measures  have  some  advantages  as  discussed 
in  Chapter  III.  A  similar  mix  of  measures,  consisting  of 
programmatic  and  ultimate  impact  indicators,  was  recommended 
by  the  Environmental  Protection  Agency  for  its  programs. 

Refinement  and  Selection  of  Final  Measures 

The  comprehensive  set  of  input  and  output  measures  was 
presented  to  the  functional  review  workshop  participants  to 
review  for  completeness  and  appropriateness.  The  entire 
list  of  candidate  measures  was  screened  to  justify  the 
acceptance  or  rejection  of  any  measure.  The  process 
involved  determining  if:  a)  the  measures  form  a  reasonable 
representation  of  the  key  DMU  variables  to  be  evaluated,  and 
b)  the  measures  form  a  realistic  description  of  the  DMU's 
activities.  The  participants,  who  have  experience  from 
various  Air  Force  levels,  provided  suggestions  for 
revisions,  deletions,  and  additions. 

Final  review  of  the  measures  list  was  accomplished  by 
an  expert  from  each  of  three  operational  levels- -base ,  Major 
Command,  and  Air  Force  Engineering  and  Services  Center.  The 
input  and  output  list  was  narrowed  down  to  those  measures 
which  best  describe  the  resources  and  critical  operations  of 
the  EPF.  Some  measures  were  refined  or  combined  with  other 
measures  to  better  reflect  the  EPF  mission  without  redun¬ 
dancy.  The  final  lists  of  inputs  and  outputs  are  provided 
at  Tables  5.2  and  5.3  respectively.  Rationale  for  dropping 
specific  inputs  and  outputs  is  explained  in  Table  5.4. 
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TABLE  5.2 


Input  Measures  Selected 


1.  Total  military  and  civilian  personnel  costs  (dollar 
value) 

2.  Total  military  and  civilian  mannours  (numoer) 

3.  Assigned  military  and  civilian  personnel  (number) 

4.  Cost  of  supplies  and  equipment  (dollar  value) 

5.  Contracted  services  expenditure  (dollar  value) 

6.  Air  pollution  sources  (numoer) 

7.  Air  quality  violations  received  (number) 

8.  Water  pollution  sources  (number) 

9.  Water  quality  violations  received  (number) 

10.  Hazardous  waste  facilities  (number) 

11.  Active  solid  waste  landfills  (number) 

12.  Solid/nazardous  waste  violations  received  (number) 

13.  Hazardous  waste  generated  (quantity) 

14.  Environmental  permits  required  (number) 

15.  Natural  resources  management  plans  and  cooperative 
agreements  (number) 


TABLE  5.3 


Output  Measures  Selected 


1.  Total  documents  reviewed  (number  and  manhours) 

2.  Total  reports  prepared  (number  ana  mannours) 

3.  Environmental  compliance  inspections  performed  (number 
and  manhours) 

4.  Responses  to  environmental  incidents  (number  and 
manhours) 

5.  Environmental  Protection  Committee  (EPC)  meetings 

(number  and  manhours) 

6.  Environmental  Impact  Analysis  Process  documents 
prepared  and  processed  (number  and  manhours) 

7.  Fines  and  penalties  paid  (dollar  amount) 

8.  Air  pollution  sources  in  compliance  (number) 

9.  Air  quality  violations  resolved  (number) 

10.  Water  pollution  sources  in  compliance  (number) 

11.  water  quality  violations  resolved  (number) 

12.  Solid/hazardous  waste  facilities  in  compliance  (number) 

13.  Solia/nazardous  waste  violations  resolved  (number) 

14.  Environmental  permits  in  compliance  (number) 

15.  Oil  and  nazaraous  substances  spills  (number) 

16.  Natural  resources  plans/agreements  updated  (numoer) 

17.  Otner  noncompliance  conditions  (number) 
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table  5.4 


Input/Output  Measures  Excluded 


A.  Measure:  Amount  of  personnel  training  and  education 
(input  2) 

Reason:  It  is  difficult  to  express  amount  of  training 

and  education  in  quantifiaole  terms. 

3.  Measure:  Overnead  and  miscellaneous  costs  (input  6) 
Reason:  The  EPF  has  minimal  expenses  in  this  category 
and  records  are  typically  not  well  kept. 

C.  Measure:  Environmental  training  sessions  conducted 
(outputs  7  and  8) 

Reason:  There  is  great  diversity  among  bases  in  wnich 

organizations  provide  training  for  environmental  issues, 
especially  for  spill  response  and  nazardous  waste 
management. 

D.  Measure:  Destruction  of  resources  due  to  pollution 
incidents  (output  9) 

Reason:  Except  for  situations  involving  wildlife  (e.g., 

fisn  kills),  human  lives,  or  facilities,  data  on  damage  is 
not  normally  estimated  or  maintained.  Also,  it  is  difficult 
to  estimate  damage  in  many  instances,  e.g.,  oil  spills 
around  tne  flight  line. 

E.  Measure:  Effective  base  population  (output  11) 

Reason:  Not  that  meaningful  compared  to  otner  measures 

when  considering  impacts  on  the  environmental  protection 
effort. 

F.  Measure:  Number  of  reporting  and  record  keeping 
discrepancies  (output  21) 

Reason:  Most  bases  do  not  maintain  data  on  this 
specific  type  of  problem. 

G.  Measure:  Past  disposal  sites  witn  investigative  studies 
completed  (input  14)  and  Past  disposal  sites  in  site 

cnar accer ization  or  remediation  pnases  (output  24) 

Reason:  The  transition  from  investigative  studies 

(Pnase  I)  to  further  investigations  and  actual  remediation 
in  Phases  II,  III,  and  IV  is  not  well  established;  many 
variables  affect  program  progress  arter  Pnase  I. 
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TABLE  5.4  continued 


H.  Measure:  Land  available  for  grazing  and  cropland 
management  (input  16)/  torest  management  (input  17)/  ana 
fisn/wildlif e/outdoor  recreation  (input  18)  ana  receipts 
from  grazing  and  cropland  management  (output  26),  forest 
management  (output  27),  ana  f ish/wildlife/outdoor  recreation 
(output  28) 

Reason:  Relating  available  acres  to  receipts  is 
difficult  because  so  many  variables  affect  the  land's 
proauctivity  (e.g.,  amount  of  land  suitable  for  designated 
purpose  and  variations  in  local  weather  and  market 
conditions) . 


Data  Collection 

A  survey  form  was  developed  to  collect  data  on  ail  the 
selected  input  and  output  measures.  The  Air  Force 
Engineering  and  Services  Center  distributed  the  survey  by 
cover  letter  to  eight  Major  Commands  on  13  May  1986.  (See 
Appendix  E  for  tne  cover  letter  ana  survey  form.)  -A  Fiscal 
Year  1985  data  base  was  used  since  this  was  the  most  recent 
reporting  cycle  ana  data  couxd  be  more  easily  estimated  for 
a  recent  year.  The  bases  which  responded  are  listed  at 
Table  5.5.  Surveys  returned  to  the  Center  were  forwarded  to 
AFIT  where  the  data  was  screened,  manipulated,  and  loaded  on 
tne  IBM- AT. 

Screening .  Tne  first  step  in  preparing  tne  survey 
information  for  entry  into  the  data  base  was  screening  for 
obvious  errors.  Several  base  environmental  coordinators 
were  called  to  either  confirm  the  data  or  ask  for  additional 


TABLE  5.5 


Installations  Returning  Surveys 


Altus  AFB 
Cannon  AFB 
Charleston  AFB 
Dover  AFB 
Ellsworth  AFB* 
Hanscora  AFB* 
Hurlburt  Field 
Langley  AFB 
Loring  AFB* 
MacDill  AFB 
McChord  AFB 
Mt.  Horae  AFB 
Norton  AFB 
Robins  AFB 
Tyndall  AFB 
Wurtsmith  AFB* 

*  Not  received 


information.  Some  of  the  data  errors  included:  a)  entering 
the  wages  as  dollars  per  hour  instead  of  yearly  totals,  b) 
entering  an  unreasonable  number,  such  as  a  base  generating 
only  one  kilogram  of  hazardous  waste,  and  c)  not  showing 
proper  correspondence  between  data  elements,  such  as  a  base 
having  air  permit  violations  but  listing  no  air  quality 
permits. 

While  several  of  the  elements  called  for  estimated 
quantities,  one  element  was  suspect  even  for  an  estimate. 

The  contracted  services  expenditure  (15)  element  ranged  from 


$7,600,000  to  $0.  Discussions  with  several  environmental 


Andrews  AFB 
Carswell  AFB* 
Columbus  AFB 
Edwards  AFB* 
Elmendorf  AFB* 

Hill  AFB 
Kelly  AFB 
Laughlin  AFB 
Lowry  AFB* 

Mather  AFB 
McGuire  AFB 
Nellis  AFB 
Patrick  AFB* 

Seymour  Johnson  AFB 
Whiteman  AFB* 


Bergstrom  AFB 
Chanute  AFB 
Davis-Monthan  AFB 
Eielson  AFB* 
England  AFB 
Holloman  AFB 
K  I  Sawyer  AFB* 
Little  Rock  AFB 
Luke  AFB 
McClellan  AFB 
Moody  AFB* 

Newark  AFS 
Reese  AFB 
Tinker  AFB 
Wright »Patt  AFB 


in  time  to  include  in  data  analysis 


respondents  included  in  the  element.  Part  of  the  problem 
resulted  from  misunderstanding  the  question  or  from 
interpreting  differently  the  meaning  of  service  contracting. 
Because  of  the  suspected  poor  reliability,  the  variable  was 
restricted  to  certain  analysis  runs. 

Except  for  the  obvious  errors,  all  data  was  considered 
correct  as  noted  in  the  assumptions  in  Chapter  I.  More 
discussion  about  the  data  will  be  in  Chapter  VII. 

Manipulation.  There  were  a  total  of  26  input  and  24 
output  survey  elements  which  potentially  could  be  used  as 
variables  in  the  analysis.  The  survey  was  developed  so  that 
the  elements  could  be  used  individually  or  combined 
appropriately  to  represent  DMU  measures.  For  example, 
depending  on  the  purpose  of  an  analysis  run,  the  number  of 
civilian  and  military  manhours  could  be  analyzed  as  two 
separate  variables  or  as  one  combined  variable  without 
hurting  measurement  validity  (i.e.,  the  variable  measures 
what  it  is  expected  to).  Air,  water,  and  solid  waste 
violations  are  examples  of  output  variables  which  can  be 
analyzed  separately  as  individual  violations  or  combined  as 
total  base  violations.  Table  5.6  lists  combined  data  base 
variables  and  their  respective  elements  from  the  survey. 

Some  variables  were  entered  as  reciprocal  values  to 
place  them  in  proper  relationship  to  other  input  or  output 
values.  When  the  models  attempt  to  optimize  a  DMU's 
efficiency,  relatively  low  values  for  inputs  and  high  values 


TABLE  5.6 


Combined  Variables 


Var iables 
(and  measures) 

Total  Cost  (11,4,5) 


Total  Pay  (II) 


Supply  and  Equipment  (14) 


Pollution  Sources 
(16,8,10,11) 


Violations  Received 
(17,9,12) 


Compliance  Status 
(08,10,12,14,17) 


Survey  Element 
(and  Number) 

Civilian  Pay  (10) 

Military  Pay  (11) 

Supplies  Costs  (12) 
Equipment  Costs  (13) 
Contracted  Services  (14) 

Civilian  Pay  (10) 

Military  Pay  (11) 

Supply  Costs  (12) 
Equipment  Costs  (13) 

Air  Pollution  (23) 

Water  Pollution  (24) 
Hazardous  Waste  (25) 

Solid  Waste  (26) 

Air  Quality  (44  &  45) 

Water  Quality  (48  &  49) 
Solid/Hazardous  Waste 
(52  &  53) 

Air  Quality  (47) 

Water  Quality  (51) 
Solid/Hazardous  Waste  (55) 
Environmental  Permits  (56) 
Other  Conditions  (59) 


for  outputs  result  in  DMUs  being  rated  more  efficient.  The 
measures  that  are  devised  so  that  high  input  and  low  output 
values  are  considered  relatively  good  need  to  be  expressed 
as  reciprocals.  For  example,  number  of  spills  is  an  output 
which  should  be  relatively  low  to  improve  DMU  efficiency. 
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Reciprocal  values  of  spills  show  DMUs  with  fewer  spills  as 
having  a  higher  numeric  value  for  that  output  measure,  which 
is  the  desired  relationship.  Reciprocal  variables  include 
Pollution  Sources,  Hazardous  Waste  Generated,  and 
Oil/Hazardous  Substances  Spills. 

The  computer  program  will  accept  zero  values  for 
outputs  but  not  as  inputs.  Substitute  values  were  created 
where  the  equivalent  of  zero  was  needed.  In  some  cases,  the 
maximum  allowed  value  of  999,999  was  entered  so  that  the 
model  would  assign  a  very  low  weight  to  the  variable,  in  a 
sense  ignoring  it.  Since  division  by  zero  is  prohibited,  a 
value  of  two  was  used  in  place  of  the  reciprocal  of  zero. 
This  compares  to  a  value  of  one  which  equals  a  reciprocal 
one;  a  value  of  two  which  equals  a  reciprocal  one*halfT  etc. 

Loading .  Operating  on  the  IBM* AT,  Productivity 
Assessment  Support  System  allows  interactive  entry  of  data 
using  menus  to  direct  user  actions.  Codes  for  each  Major 
Command  and  base  and  names  for  each  variable  were  devised, 
then  used  as  key  names  in  the  data  base.  Once  the  codes  and 
key  names  were  entered,  the  menu  asked  for  values  of  each 
variable.  The  data  base  was  corrected  using  either  the 
software  menu  or  the  system  (dBase)  file  editor. 


VI.  Analysis  and  Results 


This  Chapter  will  review  the  computer  program  execution 
and  discuss  the  test  runs,  including  any  problems  encoun¬ 
tered.  The  resulting  reports  will  be  analyzed  keeping  in 
mind  investigative  questions  2  and  3 — are  the  analytical 
models  and  DMU  measures  appropriate?  The  analysis  also  has 
implications  for  investigative  question  4  which  addresses 
the  application  of  these  models.  Further  discussions 
concern  investigative  questions  5  and  6. 


DEA/CFA  Computer  Program  Execution 

The  interactive  Productivity  Assessment  Support  System 
program  allowed  many  combinations  of  variables  to  be  used 
for  each  analysis  run.  Variables  for  all  or  part  of  the 
observations  (installations)  can  be  compared  as  long  as  no 
more  than  20  variables  are  used  and  the  observation/variable 
ratio  is  at  least  two  to  one.  To  run  the  program  the 
operator  need  only  to  specify  the  variables  and  observations 
for  the  DMU.  Once  the  computer  program  has  executed, 
various  efficiency  analysis  results  can  be  reported.  Four 
reports  used  for  this  research  and  their  descriptions  are 
listed  below  (8:26): 

1.  Efficiency  Report  -  for  each  DMU,  provides  input 
and  output  levels,  adjusted  variable  levels,  relative 
contributions,  and  shortages/excesses  if  any. 

2.  Comparison  of  Multiple  Efficiency  Analyses  - 
provides  a  summary  table  showing  CFA  and  DEA  efficiencies 
and  the  average  value  for  each  analysis.  Printouts  of  this 
report  for  each  test  are  at  Appendix  F. 


3.  List  Local  Frontier  Units  *  identifies  the 
frontier  units  and  their  characteristics  for  each 
inefficient  unit. 

4.  Summary  of  Local  Frontiers  *  produces  a  listing  of 
frontier  organizations  for  each  complete  and  incomplete 
local  frontier. 

When  there  was  an  option,  CFA  efficiencies  were 
requested  for  all  reports  to  ensure  consistency  in 
comparisons  of  DMUs.  For  Efficiency  Reports,  an  evaluation 
of  efficient  outputs  for  observed  input  levels  was 
specified.  Analysis  focused  on  outputs  since  more  control 
generally  is  exercised  over  outputs  than  inputs.  For 
example,  bases  usually  have  little  control  over  their 
budgeted  amount  of  funds  and  the  number  of  pollution  sources 
they  have. 

Initial  Analysis  Attempts 

The  first  set  of  analyses  was  devised  using 
combinations  of  variables  to  test  efficiency  in  three  areas. 
Table  6.1  lists  the  variable  combinations  for  the  initial 
analyses.  The  "overall"  test  included  most  of  the  measures 
(individually  or  in  combination)  representative  of  the 
entire  EPF  function.  The  "process"  test  included  measures 
which  consisted  of  administrative  resources  inputs  and 
office  generated  outputs.  The  "results"  test  compared 
variables  which  reflected  environmental  compliance  aspects 
of  the  EPF  program.  By  comparing  pollution  sources  (inputs) 
to  violations  (outputs),  this  test  serves  to  evaluate 
effectiveness,  rather  that  efficiency,  in  meeting  compliance 


standards . 


TABLE  6.1 


Initial  Input/Output  Combinations 


Test  1  *  Overall 


Inputs 


Outputs 


Total  Cost  (11,4) 

Manhours  (12) 

Pollution  Sources  (16,8,10,11) 
Hazardous  Waste  Generated  (113) 
Required  Permits  (114) 

Natural  Resources  Plans  (115) 
Violations  Received  (17,9,12) 


Documents  Reviewed  (01) 
Reports  Prepared  (02) 
Compliance  Inspections 
(03) 

Incident  Responses  (04) 
EPC  Activity  (05) 
Environmental  Documen- 
tation  (06) 

Violations  Resolved 
(09,11,13) 

Compliance 

(08,10,12,14,17) 

Oil  Spills  (015) 

Natural  Resources 
Plans  Updated  (016) 


Test  2  •  Process 


Total  Cost  (11,4)  Documents  Reviewed  (01) 

Manhours  (12)  Reports  Prepared  (02) 

Compliance  Inspections 
(03) 

Incident  Responses  (04) 
EPC  Activity  (05) 
Environmental  Documen¬ 
tation  (06) 


Test  3  -  Results 


Pollution  Sources( 16,8,10,11) 
Hazardous  Waste  Generated  (113) 
Required  Permits  (114) 
Violations  Received  (17,9,12) 


Violations  Resolved 
(09,11,13) 

Compliance 
(08,10,12,14,17) 
Permits  Noncompliance 
(014) 


The  overall  test  for  17  variables  was  attempted  using 
34  observations  which  had  been  received  by  24  July  1986. 

The  DEA  and  CFA  optimizations  rated  all  DMUs  as  efficient, 
that  is,  having  values  of  1.0.  Even  though  the  run  had 
twice  as  many  observations  (34)  as  variables  (17),  the 
models  were  not  able  to  make  meaningful  comparisons.  The 
diversity  of  variables  and  their  values  makes  comparison 
difficult  at  this  ratio  of  observations  to  variables.  In 
trying  to  compare  numerous  attributes,  the  models  give  each 
DMU  a  high  efficiency  rating  for  at  least  one  attribute. 

Test  2  *  Process  was  not  run  because  of  the  previously 
discussed  problem  of  too  many  variables.  With  fewer 
variables  involved.  Test  3  *  Results  was  attempted  using 
only  34  observations.  The  optimization  identified  27  DMUs 
as  efficient  with  values  of  1.0,  while  ranking  seven  DMUs  as 
less  efficient.  While  more  differentiating,  this  comparison 
also  was  not  adequate  for  a  realistic  evaluation  of 
installation  operations.  Even  with  these  shortcomings.  Test 
3  *  Results  will  be  used  as  an  example  of  how  DMUs  can  be 
compared . 

Appendix  F  contains  the  summary  listings  of  DMU 
efficiency  ratings  provided  by  DEA  and  CFA.  The  upper  bound 
columns  are  the  DEA  ratings  and  the  lower  bound  are  CFA.  If 
DEA  can  envelop  the  input  and  output  variables,  then  CFA 
analysis  is  not  necessary  and  both  upper  and  lower  bound 
values  are  identical.  if  CFA  is  not  able  to  complete 


73 


envelopment,  then  both  values  are  also  the  same.  For  Test 
3  -  Results,  most  DMU  values  are  the  same  for  these  reasons. 
But  DMU  10  is  an  example  of  a  DMU  for  which  CFA  could 
envelop  the  variables  after  DEA  could  not.  As  explained  in 
Chapter  IV,  the  envelopment  results  in  lower  values  for  the 
CFA  rating. 

TABLE  6.2 
Results  of  Test  3 


DMU 

Ratinq 

Output  Variable(s) 

10 

.2630 

Violations  Resolved,  Compliance, 
Permits  Noncompliance 

14 

.6667 

Violations  Resolved,  Compliance, 

Spills 

18 

.3282 

Violations  Resolved,  Compliance, 

Spills 

21 

.8000 

Violations  Resolved,  Permits 
Noncompliance 

57 

.7835 

Permits  Noncompliance 

63 

.8856 

Violations  Resolved,  Spills,  Permit 
Noncompliance 

Table  6.2  shows  the  seven  inefficient  DMUs  giving  the 
associated  ratings  and  the  output  variable(s)  which  largely 
contributed  to  the  inefficiency.  The  output  variables  were 
those  identified  in  the  analysis  as  having  low  "percent 
contribution  to  efficiency."  The  model  identifies  the 
percent  contribution  for  each  variable  so  that  the  variables 
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with  the  highest  percent  contribution  is  the  most 
significant  for  the  DMU.  Percent  contribution,  which 
represents  the  weighting  assigned  to  each  variable  for  a 
DMU.  As  explained  before,  the  weight  assigned  to  each 
variable  is  assigned  by  the  model  depending  on  how  a  DMU  is 
using  resources  compared  to  other  DMUs.  DMUs  may  emphasize 
a  certain  variable  or  combination  of  variables,  so  the 
significance  of  a  variable  should  only  be  determined  for  the 
DMU  being  evaluated. 

A  good  way  to  illustrate  this  is  to  compare  two  DMUs 
with  similar  efficiencies  as  shown  in  Table  6.3. 

TABLE  6.3 

Comparison  of  Variable  Percent  Contribution 


Output  Contribution  to  Efficiency 


DMU  3 

DMU  50 

Violations 

Resolved  (09,11,13) 

39.818 

39.785 

Compliance 

(08,10,12,14,17) 

21.200 

00.000 

Oil  Spills 

(015) 

38.906 

60.216 

Permits  Noncompliance  (014) 

00.000 

00.000 

99.924  100.001 


By  having  successful  efforts  in  the  first  three 
outputs,  DMU  3  was  considered  as  efficient  as  DMU  50  which 
was  efficient  in  violations  resolved  and  preventing  oil 
spills.  DMU  3  may  want  to  emphasize  performance  in  the  last 
output  to  slightly  improve  its  efficiency  rating  if  the 
improvement  can  be  done  without  excessive  use  of  resources. 
Even  though  relative  efficiency  would  not  go  up,  DMU  50 
could  work  to  improve  its  low  outputs. 

A  noteworthy  characteristic  of  the  Results  test  was 
that  19  of  34  observations  had  efficiency  ratings  of  1.0  due 
*-o  the  contribution  of  just  one  variable.  As  discussed 
before,  this  poor  differentiation  resulted  from  the  models 
finding  most  of  the  DMUs  efficient  in  at  least  one  variable 
in  relation  to  other  DMUs.  Because  the  observations  did  not 
more  than  double  the  number  of  variables,  many  DMUs  had 
characteristics  in  which  they  could  excel  and  thus  be  rated 
efficient.  One  method  to  alleviate  the  poor  differentiation 
is  to  increase  the  observations  by  including  more  DMUs  or  by 
examining  several  DMUs  over  many  time  periods. 

The  Results  test  points  out  other  interesting  aspects 
of  the  analysis  for  EPFs.  DMU  57  had  only  one  variable 
(Permits  Noncompliance)  causing  inefficiency,  while  the 
other  DMUs  had  more  than  one.  The  value  level  for  Permits 
Noncompliance  was  0.0  which  represents  total  compliance. 
Having  a  low  output  level  for  good  performance  seems 
illogical,  but  the  justification  relates  to  a  comparison 


between  inputs  and  outputs.  In  this  case,  the  DHU  had  no 
required  permits,  which  is  given  an  input  value  of  999,999, 


representing  relatively  unimportant  inputs.  Therefore, 
neither  input  nor  output  for  Permit  Noncompliance 
contributed  to  the  efficiency  of  DMU  57.  Similar  logic 
applies  to  other  situations.  For  example,  a  DMU  with  five 
permits  but  with  only  four  in  compliance  would  have  an 
input/output  ratio  of  4/5.  A  unit  with  a  totally  efficient 
variable  would  have  five  out  of  five  permits  in  compliance 
for  a  ratio  of  one.  Comparing  one  input  to  one  output  is 
only  part  of  the  models  evaluation,  since  overall  DMU 
efficiency  is  computed  using  comparisons  of  many  variables 
and  DMUs . 

Examining  DMU  57  and  G,  rated  78.322  and  88.600 
respectively,  shows  how  the  models  handle  variables  with 
zero  input  and  output.  Both  DMUs  had  0.0  or  low  values  for 
two  or  three  output  variables,  indicating  they  had  few 
sources  in  compliance.  However,  their  input  levels  of  zero 
indicate  they  had  few,  if  any,  sources  which  have  to  be  in 
compliance.  Thus,  while  the  DMUs  were  effective  at  avoiding 
compliance  problems,  they  were  rated  as  not  being  efficient 
in  doing  so.  When  evaluating  the  analysis  results  and 
making  recommended  changes,  managers  should  consider  that 
low  efficiency  ratings  may  not  mean  poor  EPF  compliance. 

Once  the  inefficient  variables  or  DMUs  have  been 
identified,  the  models  determine  how  much  change  is  needed 
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to  reach  maximum  efficiency.  In  the  Efficiency  Report,  the 
models  provide  an  adjusted  input/output  level  which  would 
bring  inefficient  DMUs  up  to  a  1.0  rating.  These  levels  are 
determined  by  multiplying  the  variables’  values  by  a 
"multiplier  for  adjusting  output  levels."  Managers  could 
improve  efficiency  by  attempting  to  reach  these  values 
without  also  raising  resource  consumption.  For  example,  DMU 
10  would  need  to  improve  compliance  or  reduce  the  number  of 
spills  as  shown  below. 


Output 

Adjusted 

Variable 

Level 

Multiplier 

Output 

Compliance 

167.0 

3.8023 

634.981 

Oil  Spills 

O 

• 

in 

It 

19.011 

In  practical  terras,  DMU  10  would  need  to  reduce  spills  from 
two  to  zero  or  noncompliance  situations  from  six  to  two  or 
less . 

Analysis  Results  Using  Variable  Combinations 

Considering  the  poor  success  of  running  analyses  using 
only  34  observations,  the  analysis  shifted  to  tests  using 
fewer  variables.  Groupings  of  fewer  than  nine  variables 
were  devised  to  provide  results  which  better  evaluate  EPF 
operations  and  the  capabilities  of  the  models  to  evaluate 
EPFs.  Attempts  to  place  variables  into  input  or  output 
combinations  which  have  similar  attributes,  such  as  costs, 
manhours,  or  compliance  issues  were  made.  The  new 
combinations  are  listed  in  Table  6.4. 


One  result  of  these  analyses  was  immediately  obvious. 
The  reduced  number  of  variables,  and  possibly  the  grouping 


of  more  comparable  variables,  resulted  in  better  envelopment 
of  DMUs  and  fewer  efficient  DMUs.  EPFs  could  be  better 
evaluated  because  of  better  DMU  differentiation.  Further 
detailed  evaluations  are  in  the  following  sections. 


TABLE  6.4 

Alternative  Variable  Combinations 


Inputs 


Manhours 


Manhours 

Personnel  Assigned 


Total  Pay 

Supply  and  Equipment 
Contract  Costs 


Outputs 


Manhours  1 

Documents  Reviewed 
Reports  Prepared 
Compliance  Inspections 
Responses 
EPC  Activity 
Environmental  Document 

Manhours  2 

Documents  Reviewed 
Reports  Prepared 
Compliance  Inspections 
Responses 
EPC  Activity 
Environmental  Document 

Costs  1 


Documents  Reviewed 
Reports  Prepared 
Compliance  Inspections 
Responses 
EPC  Activity 
Environmental  Document 


TABLE  6.4  Continued 


Costs  2 


Total  Pay 

Supply  and  Equipment 
Contract  Costs 


Total  Cost 


Violations  Resolved 

Compliance 

Oil  Spills 


Costs  3 


Documents  Reviewed 
Reports  Prepared 
Compliance  Inspections 
Responses 
EPC  Activity 
Environmental  Document. 


Costs  4 


Total  Cost 


Air  Sources 
Air  Violations 


Violations  Resolved 

Compliance 

Oil  Spills 


Air  Quality 


Air  Noncompliance 
Violations  Resolved 


Water  Quality 


Water  Source 
Water  Violations 


Water  Noncompliance 
Violations  Resolved 


Solid  Waste 


Solid  Waste  Facilities 
Hazardous  Waste  Generated 
Solid  Waste  Violations 


Solid  Waste  Noncomp. 
Oil  Spills 
Violations  Resolved 


Violations 


Violations  Received 


Pollution  Sources 
Required  Permits 


Violations  resolved 


Compliance 


Compliance 

Permit  Noncompliance 


Manhours  1.  Manhours  1  determined  how  efficient  DMUs 
were  in  completing  process  objectives  in  relation  to 
manhours,  a  key  resource  in  the  mostly  white-collar  EPF. 
DEA/CFA  evaluated  five  DMUs  as  efficient  (1.0),  while  the 
lowest  unit  was  DMU  22  with  .0380.  A  comparison  of  these 
extremes  shows  DMU  10  completed  909  units  of  output  with 
1859  manhours,  while  DMU  22  completed  only  61  units  of 
output  with  13,566  manhours.  Managers  should  keep  in  mind 
that  only  efficiency  in  using  manhours  is  being  evaluated 
and  DMU  22  output  may  have  been  concentrated  in  some  other 
area  such  as  compliance.  DMU  10,  8,  18,  21,  and  66  had 
ratings  of  1.0  and  only  one  or  two  variables  contributed  to 
the  efficiency.  For  example,  DMU  21  had  16  EPC  meetings 
which  contributed  100  percent  of  the  efficiency.  Other 
variables  (such  as  Inspections)  had  high  values  but  were  not 
needed  to  contribute  to  DMU  efficiency.  The  same  number  of 
EPC  meetings  (16)  was  not  enough  to  rank  DMU  23  higher  than 
41.357  percent  efficient  since  values  of  its  input  variables 
were  higher.  Surprisingly,  the  number  of  EPC  meetings  for 
DMU  1  was  only  the  minimum  number  (four  per  year)  required 
by  Air  Force  regulations,  yet  it  received  12.165  percent 
contribution  to  efficiency. 

Conversely,  not  one  DMU  had  all  output  variables 
contributing  to  efficiency.  Disparities  in  contribution  may 
result  from  the  differences  in  variable  characteristics  and 
values,  so  that  some  variables  continue  to  dominate 
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efficiency.  These  results  from  Manhours  1  show  how  the 
models  perform  complex  comparisons  no  manager  could  perform 
manually. 

The  results  show  that  some  DMUs  dominate  in  certain 
areas.  The  Reports  Prepared  variable  had  zero  contribution 
to  efficiency  in  22  of  34  DMUS.  Apparently,  a  relatively 
high  value  of  1200  for  DMU  I  resulted  in  other  DMUs 
receiving  low  ratings  for  that  variable.  Even  that  high 
value  only  placed  DMU  I's  efficiency  at  .525  percent. 
Managers  may  want  to  consider  dropping  extreme  variable 
values  to  see  how  DMUs  compare  to  units  with  more  similar 
magnitude. 

Manhours  2.  Manhours  2  test  added  the  input  variable. 
Personnel  Assigned,  keeping  output  variables  unchanged. 
Except  for  four  ratings  which  essentially  remained  constant, 
all  ratings  went  higher  or  could  go  no  higher  than  their 
existing  rating  of  1.0.  The  addition  of  a  variable  may  have 
given  more  areas  for  DMUs  to  excel  and  be  efficient  as 
discussed  under  the  Results  test.  Of  the  four  DMUs  which 
changed  little,  DMU  20  remained  identical  because  the 
Personnel  Assigned  variable  contributed  zero  to  input 
efficiency.  Apparently  the  personnel  assigned  related  to 
the  other  DMUs  such  that  the  other  variables  kept  their  same 
relationships.  The  similar  alignment  of  ratings  for 
Manhours  1  and  Manhours  2  analyses  was  expected  since  the 
number  of  personnel  (the  new  variable)  is  closely  related  to 
manhours . 
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Costs  1.  Costs  1  tested  DMU  efficiency  in  using  their 


financial  resources  as  input  to  produce  administrative 
output  (e.g.,  paperwork).  The  models  identified  12  DMUs  as 
efficient.  Once  again,  the  efficient  units  received  all  of 
their  rating  from  one  variable.  Managers  need  to  be  wary 
since  slips  in  this  variable  could  hurt  the  entire  rating. 
The  variations  of  DMU  from  Manhours  tests  to  Costs  1  tests 
was  dramatic  except  for  some  of  the  units  rated  as 
efficient.  Of  the  six  DMUs  which  remained  efficient,  four 
had  one  or  two  variables  which  contributed  most  of  the 
efficiency  in  both  tests.  Regardless  of  this  point,  the  six 
DMUs  appear  to  have  a  good  efficient  combination  of  process 
output  variables. 

Costs  2.  The  Costs  2  approach  was  more  results 
oriented  to  see  how  efficient  DMUs  were  in  using  money  to 
stay  in  compliance.  Many  of  the- efficient  DMUs  were 
efficient  under  Costs  1  and  2.  Since  the  output  variables 
changed,  the  efficient  DMUs  appeared  to  rely  on  the  input 
variables  to  remain  efficient.  This  test  was  the  first 
analysis  where  all  output  variables  contributed  in  some  way 
to  DMU  efficiency.  The  reduced  number  (6)  of  variables  or 
the  variable  characteristics  may  have  caused  more  even 
distribution  of  efficiency  contribution. 

Costs  3.  A  noticeable  difference  occurred  under  Costs 
3  where  Total  Cost  was  the  only  input  variable.  (Total  Cost 
combines  the  Total  Pay  and  Supply/Equipment  Cost  variables, 


but  drops  the  Contract  Costs  variable.)  Only  three  DMUs 
remained  efficient  (a  drop  from  11)  and  all  but  two  DMUs 
dropped  in  efficiency.  These  lower  ratings  probably  more 
accurately  reflect  the  true  DMU  efficiencies.  As  mentioned 
in  Chapter  IV,  Contract  Costs  had  extreme  variations  in 
values  which  may  have  disrupted  the  evaluation.  For  example 
under  Costs  1,  Contract  Costs  contributed  52  percent  of  DMU 
50' s  input  efficiency  by  being  a  relatively  low  value.  DMU 
50  was  rated  76  percent  efficient  under  Costs  1,  but  with 
Contract  Costs  removed  under  Costs  3,  the  DMU  was  only  12 
percent  efficient.  The  high  values  for  just  a  few  DMUs  may 
have  made  the  remaining  low  values  look  much  better  and 
distorted  their  efficiency.  Most  other  DMUs  with  dramatic 
drops  in  efficiency  exhibited  this  same  change. 

The  two  DMUs  that  exhibited  negligible  change  point 
out  an  interesting  aspect  of  the  analysis.  The  variables 
Supply/Equipment  and  Contract  Costs  were  zero  values 
(entered  as  999999).  The  model  ignored  these  two  variables 
(giving  them  zero  contribution)  and  used  only  total  pay  for 
comparison.  In  these  cases.  Total  Pay  was  the  same  value  as 
Total  Cost  for  the  Costs  1  test. 

Costs  4_.  Costs  4  compared  Total  Cost  to  compliance 
variables  and  showed  a  familiar  pattern  of  the  DMU 
efficiency  ratings  dropping  from  the  Costs  2  test.  Only  DMU 
18  with  a  perfect  compliance  record  and  DMU  61  with  35 
violations  resolved  were  able  to  rate  1.0. 


Table  6.5  ranks  the  DMUs  in  the  two  Manhours  and  four 
Costs  tests  according  to  their  efficiency  ratings.  A 
correlation  between  DMU  rankings  in  Manpower  and  Costs  tests 
might  be  expected  since  total  pay  (Costs)  is  linked  to 
manhours  and  personnel  assigned  (Manhours).  Contracting 
Costs,  which  show  a  high  degree  of  variability  among  DMUs, 
probably  account  for  much  of  the  efficiency  differences. 
Correlation  analyses  tend  to  corroborate  this  idea.  The 
correlation  between  DMU  rankings  in  Manhours  1  and  Costs  1 
(includes  Contract  Cost)  was  only  r  =  0.380.  The  correla¬ 
tion  for  Manhours  1  and  Costs  3  (excludes  Contract  Cost)  was 
r  =  0.837,  which  is  considered  over  99  percent  significant 
for  34  samples.  Combining  Manhours  1  and  Costs  3  into  one 
test  because  of  their  similarities  could  produce  as  good  or 
better  results  with  less  redundancy. 

Compliance  Tests 

This  section  discusses  analyses  of  variables  which  are 
results  oriented  and  indicators  of  environmental  compliance. 

Air  Quality.  The  most  obvious  result  of  this  test  was 
that  most  of  the  units  were  rated  efficient  (1.0)  or 
inefficient  (0.0).  A  zero  efficiency  rating  is  unusual  for 
DEA  and  CFA ,  but  is  understandable  considering  the  way  the 
tests  were  set  up.  Efficient  units  had  output  equal  to 
input — they  were  in  compliance  and  resolved  all  violations. 
The  inefficient  units  had  zero  inputs  and  outputs — they  had 
not  air  pollution  sources  or  violations.  Basically,  the 
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RESOLUTION  TEST 


TABLE  6.5 


DMU 

Rankings 

for  Manhours 

and 

Costs  Tests 

DMU 

Manl 

Man2 

Costl 

Cost2 

Cost3 

Cost4 

1 

2 

1 

14 

1 

9 

3 

2 

9 

6 

12 

2 

15 

9 

3 

21 

22 

7 

1 

22 

8 

6 

18 

5 

20 

4 

11 

4 

8 

1 

1 

1 

13 

7 

13 

10 

1 

1 

1 

1 

1 

11 

12 

23 

25 

1 

9 

30 

27 

14 

25 

27 

16 

10 

16 

23 

16 

7 

7 

15 

14 

18 

26 

18 

1 

1 

3 

1 

2 

1 

19 

12 

2 

13 

17 

3 

27 

20 

5 

9 

7 

15 

8 

15 

21 

1 

1 

1 

1 

1 

6 

22 

2\ 

28 

22 

12 

31 

25 

23 

8 

10 

18 

6 

12 

18 

50 

27 

24 

4 

1 

27 

16 

51 

26 

23 

19 

11 

26 

24 

53 

13 

12 

1 

1 

13 

7 

54 

3 

4 

1 

5 

4 

5 

55 

10 

11 

11 

1 

10 

12 

56 

15 

21 

17 

19 

24 

26 

57 

11 

15 

10 

8 

25 

14 

58 

16 

16 

1 

1 

19 

1 

61 

20 

20 

21 

16 

29 

19 

63 

14 

14 

1 

7 

21 

20 

64 

24 

26 

8 

22 

25 

17 

66 

1 

1 

1 

1 

1 

2 

70 

17 

13 

9 

18 

20 

28 

71 

14 

8 

1 

1 

14 

21 

72 

16 

18 

6 

23 

28 

31 

73 

4 

3 

1 

1 

5 

10 

74 

22 

17 

5 

20 

23 

29 

75 

6 

1 

1 

21 

6 

30 

76 

19 

19 

2 

3 

17 

22 
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inefficient  ratings  should  be  ignored  since  the  DMUs  did  not 
need  to  be  in  the  comparison  anyway.  Including  DMUs  with 
zero  values  may  have  distorted  the  results  by  making  zero 
the  baseline  input  and  output  comparison  value  for  other 
units. 

Water  Quality.  This  test  had  characteristics  similar 
to  Air  Quality. 

Solid  Waste  Quality.  The  Solid  Waste  test  showed  more 
differentiation  than  the  Air  and  Water  tests  probably 
because  of  the  additional  input  and  output  variables. 
Hazardous  Waste  Generated  and  Oil  Spills.  These  variables 
appeared  to  have  other  effects.  Evaluating  solid  waste 
compliance  performance  in  relation  to  the  amount  of 
hazardous  waste  generated  placed  many  larger  or  industrial 
installations  at  close  to  1.0  efficiency  ratings.  In 
comparison,  one  Solid  Waste  Quality  test  run  without  the 
Hazardous  Waste  Generated  variable  rated  the  large  DMUs 
consistently  lower.  These  large  units'  compliance  was 
relatively  not  as  good  but  their  compliance  problems  were 
greater . 

Violations .  This  test  determined  how  efficient  DMUs 
were  in  resolving  existing  pollution  violations.  Eleven 
units  were  rated  efficient  (1.0)  while  eight  DMUs  were  rated 
inefficient  (0.0).  In  the  case  of  inefficient  units,  the 
results  must  be  carefully  interpreted.  For  most  of  these 
units,  the  DMUs  had  no  violations  (zero  input)  and  therefore 


nothing  to  resolve.  The  model  ranked  these  as  inefficient, 
but  managers  should  disregard  the  DMUs'  ratings  for  this 
test. 

However,  DMU  10  had  one  violation  which  was  not 
resolved,  resulting  in  an  inefficient  rating.  A  low  rating 
would  be  expected  and  the  manager  should  be  able  to 
recognize  this  relationship. 

Compliance .  The  comparison  for  compliance  resulted  in 
much  different  results  and  greater  DMU  differentiation  than 
violations  because  of  the  addition  of  one  variable  and  the 
use  of  reciprocal  values. 

In  an  attempt  to  improve  test  runs  by  reducing  the 
number  of  variables,  the  other  extreme  of  too  few  variables 
was  reached.  In  the  cases  of  Air,  Water,  and  Solid  Waste, 
and  Violations,  the  problem  was  accentuated  by  many  zero 
value  variables.  The  Violations  test  had  only  one  input  and 
one  output,  usually  with  single  digit  numbers.  These 
efficiency  ratings  were  predictable  and  the  ratios  easily 
calculated  by  hand.  So  for  some  tests,  the  models' 
capabilities  were  not  adequately  used.  The  compliance  tests 
need  to  be  combined  to  adequately  differentiate  among  DMUs 
so  that  differences  in  efficiency  are  discernable.  Of 
course,  the  number  of  observations  for  this  research 
restricted  the  combination  attempts. 

Reducing  the  number  of  variables  in  the  Compliance 
test3  illustrated  how  frontiers  are  related  to  variables. 


Efficient  units  form  the  efficiency  frontier  for  inefficient 
units  with  similar  mixes.  They  have  similar  input  and 
. tput  ratios  (not  magnitudes).  Efficient  frontier  units 
determine  the  input  and  output  weights  for  bringing  other 
units  up  to  efficiency  (8:52). 

Solid  Waste  Quality  was  one  of  the  few  tests  in  which 
"complete  frontiers"  were  formed  to  totally  envelop  the 
inefficient  DMUs.  One  of  the  three  frontiers  was  composed 
of  DMUs  20,  51 f  and  G.  To  become  efficient  and  reach  the 
frontier,  DMU  10  would  need  0.33  more  efficiency  to  reach 
DMU  20  or  51  and  no  more  efficiency  to  reach  DMU  G. 

Fisher  observed  that  the  number  of  variables  affects 
the  number  of  units  reaching  efficiency  (23:94).  Fewer 
variables  used  in  the  tests  reduces  the  size  of  the 
efficiency  frontier  and  thus  the  number  of  units  which  can 
reach  the  efficiency  frontier.  In  looking  at  Table  6.6, 
this  relationship  held  true  for  the  some  of  the  Process 
tests,  but  not  for  the  Results  tests.  The  disparity 
probably  resulted  from  the  problems  with  the  test 
arrangements  discussed  before. 

Use  of  the  Reports 

Investigative  Question  4  asked  about  the  use  of 
DEA/CFA  results  by  EPF  managers.  Productivity  Assessment 
Support  System  generates  numerous  reports,  three  of  which 
were  used  in  this  Chapter.  Managers  can  use  these  reports 
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TABLE  6.6 


Comparison 

of  Variables 

to  Efficiency 

Test 

Number  of 
Variables 

Ratio  of 
DMUS  to 
Variables 

Number 

Efficient 

Percent 

Efficient 

Costs  1 

9 

3.78 

12 

35 

Manhours 

2 

8 

4.25 

7 

21 

Manhours 

1 

7 

4.86 

5 

15 

Water 

4 

8.50 

14 

41 

Violations 

2 

17.00 

11 

32 

to  evaluate  the  efficiency  (e.g.,  manhours  and  costs)  and 
effectiveness  (e.g.,  violations  and  compliance)  of  their 
DMUs  and  find  inefficient  areas.  But  managers  are 
responsible  for  correctly  interpreting  the  results,  keeping 
in  mind  what  different  variable  combinations  can  do  to  the 
test.  Each  test  must  be  interpreted  separately,  then  in 
relationship  to  other  tests,  before  taking  management 
actions. 

The  reports  provide  "adjusted  output  levels"  that 
managers  should  strive  to  meet  to  make  DMUs  more  efficient. 
The  higher  the  percent  contribution,  the  more  significance 
the  model  is  putting  on  a  variable.  Managers  should  strive 
to  improve  variables  with  lower  percentages  since  they  are 
contributing  the  least  to  efficiency.  For  environmental 


compliance,  the  manager  would  need  to  ensure  that  attempts 
to  improve  efficiency  did  not  result  in  lowering  effective- 
ess.  For  example,  producing  more  reports  may  improve 
efficiency  but  may  have  increased  noncorapliance  as  a 
concomitant  outcome. 

Productivity  Assessment  Support  System  reports  provide 
listings  of  frontier  units  which  are  used  as  comparison  for 
other  less  efficient  DMUs.  The  DMUs  and  the  comparison 
frontier  units  have  similar  mixes  of  input  and  output 
values.  Managers  can  study  the  efficient  units  to  make 
changes  in  their  DMUs,  modeling  operations  after  the 
efficient  units.  Once  again,  managers  would  need  to  be 
careful  not  to  overemphasize  some  measures  at  the  expense  of 
others  which  could  result  in  serious  violations.  Also, 
managers  may  feel  that  high  efficiency  contributed  by  a  few 
measures  is  not  a  desirable  situation.  More  emphasis  may 
need  to  be  placed  on  the  other  variables  to  have  a  better 
balanced  program. 

Managers  do  not  have  to  be  passive  in  conducting 
analyses.  Following  the  methodology  in  this  research,  they 
can  determine  the  number  and  types  of  variables  in  a 
particular  analysis.  As  was  seen  earlier,  comparisons  of 
similar  measures  provide  different  results  from  dissimilar 
measures.  Also,  the  number  of  variables  affects  the  number 
of  efficient  units  identified.  Fewer  variables  usually 
results  in  fewer  units  reaching  the  efficiency  frontier. 


Another  step  managers  can  take  is  to  assign  weights  to  the 
variables  rather  than  allowing  the  models  to  determine  them. 
Air  Force  managers  assigned  weights  for  Donovan's  research 
(22)  with  positive  results. 

Finally,  managers  can  operate  Option  4,  Explore 
Alternatives,  of  the  Productivity  Assessment  Support  System 
which  provides  "what  if"  capabilities.  By  manipulating 
particular  variables,  managers  can  see  the  overall  effect  on 
DMO  efficiency.  The  information  allows  managers  to  evaluate 
tradeoffs  between  inputs  and  outputs  prior  to  actually 
making  changes.  Managers  must  interpret  the  results  for 
their  DMUs  taking  into  consideration  unique  characteristics. 

Headquarters  Air  Force  and  Major  Commands  would  have 
use  for  these  analyses  from  different  perspectives.  Major 
Commands  may  want  to  compare  base  programs  to  see  if  there 
are  inefficient  units  or  if  there  are  efficient  units  which 
.can  serve  as  models.  One  important  evaluation  would  be  to 
ensure  that  base  DMUs  place  proper  operational  emphasis  on 
results  measures.  Headquarters  Air  Force  would  be  less 
interested  in  individual  base  performance  than  in  agency 
ability  to  reach  overall  program  success.  Air  Staff  could 
run  the  model  using  the  data  available  from  the  Defense 
Environmental  Status  Report  as  was  the  case  in  this  research 
for  some  of  the  Results  tests.  Their  evaluation  would  find 
poor  performance  areas,  possibly  indicating  needed  program 
changes . 


Implementation  of  the  Models 


Investigative  question  5  pursues  the  idea  of  how 
analytical  models  can  be  implemented  in  the  Air  Force  to 
allow  managers  to  study  EPF  performance.  The  models  of 
choice  are  DEA  and  CFA  which  must  be  computerized  to  be 
useful  to  the  manager.  Running  these  models  with  several 
variables  for  several  observations  must  rely  on  computer 
computational  power.  Even  the  IBM-AT  microcomputer's  slow 
processing  time  would  be  too  time  consuming  for  managers.  A 
minicomputer  or  remote  hookup  to  a  computer  mainframe  would 
be  necessary  for  practical  office  application. 

Each  manager  would  require  access  to  a  computer 
terminal  since  these  programs  work  best  interactively. 

Using  their  experience  with  a  particular  installation, 
managers  would  want  to  be  able  to  alter  the  list  of  measures 
and  possibly  assign  weights  to  them.  The  models  need  to  be 
run  regularly  to  observe  trends  in  performance,  and  "what 
if?"  analyses  require  direct  terminal  interface.  Depending 
on  the  operating  system  and  data  base  used,  data  could  be 
entered  for  the  measures  and  the  model  run  nearly  on  a  real 
time  basis.  This  research  required  too  much  data  manipu- 
ation  to  allow  realistic  application  to  the  office  environ- 
ent.  Either  operating  system  features  or  program  software 
changes  could  be  made  to  automate  the  data  manipulation 
steps.  In  this  way,  managers  could  enter  data  directly  from 
the  reports  or  surveys. 
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Improvements  to  the  Productivity  Assessment  Support 
System  software  would  benefit  practical  use  of  the  models. 
While  the  System  is  touted  as  being  user  friendly,  the 
author  encountered  numerous  obstacles  to  free  interaction. 
After  going  several  levels  into  the  program  menu  to  perform 
an  operation,  the  operator  must  return  to  the  upper  level  to 
change  an  item  to  be  evaluated.  The  edit  function  for 
changing  variables  was  tedious,  requiring  a  complete 
hierarchy  of  identification  for  each  variable  to  be  entered 
for  each  change.  Some  errors  in  processing  occurred:  a) 

The  program  often  could  not  handle  values  that  overflowed 
field  length,  causing  termination  rather  than  an  error 
statement  and  continuation  of  processing;  b)  The  edit 
function  often  found  the  wrong  variable  from  that  which  was 
requested;  and  c)  Some  subroutines  would  not  find  the 
correct  variables  when  an  identification  number  above  10  was 
used. 

All  Air  Force  units  should  be  aware  of  the  models' 
current  limitation  of  20  variables  and  the  two  to  one 
observations  to  measures  ratio.  Air  Staff  would  have 
numerous  bases  available  for  comparison,  but  Major  Commands 
and  installations  may  need  a  series  of  comparisons. 
Evaluations  over  several  time  periods,  as  done  by  Donovan 
(22),  would  increase  the  number  of  observations  and  provide 
other  benefits,  such  as  indicating  performance  trends. 
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Implications  for  Operational  Changes 

This  section  addresses  investigative  question  6  to 
determine  if  the  analysis  results  in  any  suggestions  for 
operational  use  or  improvement.  The  data  analyses  performed 
indicate  that  some  installations  are  more  efficient  at  using 
resources,  some  are  more  effective  at  being  in  compliance 
and  some  perform  well  in  both  categories.  By  comparing  DMU 
rankings  in  these  categories,  some  indication  of  installa¬ 
tion  performance  necessary  for  compliance  may  be  indicated. 
Table  6.7  lists  the  34  DMUs  and  their  average  rankings  in 
the  various  tests  performed. 


TABLE  6.7 

DMU  Rankings  for  Process  and  Compliance  Tests 


DMU 

Process 

Rank 

Compliance 

Rank 

DMU 

Process 

Rank 

Compliance 

Rank 

1 

7 

12 

2 

10 

22 

3 

19 

5 

6 

14 

14 

8 

8 

15 

10 

4 

27 

12 

25 

13 

14 

27 

17 

16 

22 

18 

18 

2 

30 

19 

17 

28 

20 

13 

10 

21 

3 

24 

22 

31 

6 

23 

16 

1 

50 

23 

7 

51 

30 

9 

53 

9 

19 

54 

5 

26 

55 

12 

21 

56 

28 

16 

57 

21 

29 

58 

11 

3 

61 

29 

25 

63 

18 

12 

64 

28 

11 

66 

1 

8 

70 

24 

23 

71 

13 

15 

72 

28 

20 

73 

6 

2 

74 

26 

23 

7  5 

15 

4 

76 

20 

18 

The  average  DMU  ranking  for  Process  tests  was  compared 
to  the  ranking  for  Compliance  tests  using  correlation 
analysis.  With  r  *  -0.097  between  the  Process  and 
Compliance  rankings,  there  are  few  implications  of  how 
organizations'  efficiency  can  help  their  compliance 
effectiveness.  The  lack  of  correlation  may  indicate  that 
DMUs  with  compliance  problems  are  concentrating  on 
processing  violations  rather  than  paperwork.  Some  of  the 
DMUs  with  high  or  low  rankings  had  the  following 
characteristics : 

DMU  18  -  High  efficiency,  low  compliance.  It  had  very 
few  compliance  problems  or  pollution  sources,  so  its  efforts 
were  apparently  focused  on  administrative  matters. 

DMU  22  -  Low  efficiency,  high  compliance.  The 
compliance  record  resulted  from  a  larger  number  of  sources 
which  were  in  mostly  in  compliance.  Very  few  documents  were 
produced . 

DMU  23  -  Medium  efficiency,  highest  compliance.  A 
large  number  of  sources  were  in  compliance.  But  a  large 
staff  to  stay  in  compliance  apparently  reduced  efficiency. 

DMU  57  -  Low  efficiency,  low  compliance.  The  unit  had 
few  sources  or  violations,  but  had  high  costs.  The  high 
Contract  Costs  may  have  indicated  it  was  working  on  other 
compliance  efforts,  such  as  the  Installation  Restoration 
Program,  not  evaluated  by  this  research. 


DMCJ  61  -  Low  efficiency,  low  compliance.  The  low 
marks  apparently  came  from  few  reports  produced  and  a  large 
number  of  noncorapliances. 

DMO  66  -  Hignest  efficiency,  high  effectiveness.  Tnis 
model  unit  had  very  low  costs  and  mannours,  but  attained 
good  compliance  tor  many  sources. 

DMO  73  -  Similar  characteristics  to  DMO  66. 

Tnese  assessments  are  not  totally  valid  characterizations  of 
the  actual  organizational  performance.  But  using  analysis 
techniques  similar  to  the  ones  presented  in  this  research, 
the  Air  Force  could  use  efficient  and  effective  DMUs  as 
models  for  otner  units. 

Concentrating  on  increasing  outputs  is  one  of  tne  few 
ways  EPFs  can  expect  to  increase  efficiency  or  effective¬ 
ness.  Unfortunately,  "more  for  less"  does  not  set  well  with 
EPFs,  since  they  cannot  be  expected  to  keep  up  witn 
sxyrocketing  environmental  compliance  requirements  witn 
existing  resources.  EPFs  will  probably  not  be  getting  more 
efficient  as  tney  are  constantly  trying  to  increase 
resources.  The  increased’  resources  could  result  in  more 
output  helping  to  maintain  efficiency  and  improve 
effectiveness  levels.  However,  reducing  inputs,  such  as 
sources  of  pollution,  snould  not  oe  slighted.  Hazardous 
waste  reduction  is  an  excellent  method  to  reduce  costs  and 
pollution  potential. 
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VII.  Conclusions  and  Recommendations 


Introduction 

This  chapter  presents  conclusions  about  the 
development  of  Environmental  Planning  Function  (EPF) 
measures  and  the  use  of  these  measures  in  the  analysis 
model.  Conclusions  are  made  for  each  of  the  research 
objectives  presented  in  Chapter  I.  Chapter  VI  discussed 
ideas  about  the  use  of  the  model  by  managers  and  its 
applications  in  the  Air  Force.  This  chapter  makes 
recommendations  about  the  feasibility  of  implementation, 
taking  into  account  the  limitations  of  the  model  and 
computer  program.  Additional  recommendations  relate  to 
other  findings  made  during  the  research. 

Conclusions 

The  main  objectives  of  this  research  were  to  develop 
measurement  criteria  for  the  EPF,  find  a  model  to  evaluate 
these  measures,  and  test  the  model  for  possible  use  at  Air 
Force  installations.  Specific  conclusions  about  the  success 
in  meeting  the  research  objectives  are  discussed  below. 

Objective  _1.  The  first  objective  was  to  determine  the 
measures  which  define  EPF  operations.  Chapter  ill  described 
important  aspects  of  public  agency  productivity  and 
suggested  possible  environmental  agency  input  and  output 
measures.  Chapter  IV  proposed  a  methodology  for  selecting 
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measures,  then  Chapter  V  made  the  actual  selection.  The 
following  are  conclusions  regarding  Objective  Is 

1.  A  detailed  set  of  input  and  output  measures  for 
environmental  agencies  can  be  developed,  even  though  public 
service  organizations  are  difficult  to  characterize.  More 
measures  are  usually  needed  for  public  agencies  than  for 
private  firms  (10:273;  25:385).  Hence,  the  EPF  was 
described  in  quantitative  terras  using  32  process  or  results 
oriented  measures.  Even  so,  the  measures  did  not  account 
for  all  of  the  Environmental  and  Contract  Planning  section 
and  all  programs,  such  as  Installation  Restoration.  The 
measures  developed  during  this  research  already  need 
updating  as  program  emphasis  shifts  from  traditional  areas 
(e.g.,  air  and  water  pollution)  to  the  Environmental 
Compliance  and  Management  Program  and  other  new  programs. 
Developing  measures  is  an  iterative,  dynamic  process. 

2.  Few  public  or  private  sector  organizations  are 
available  as  prototypes  for  developing  EPF  measures. 

Chapter  III  reviewed  literature  on  productivity  measurement 
in  various  agencies,  but  few  studies  were  found  specifically 
on  environmental  agency  measurement.  The  scant  research  was 
not  surprising  because  of  the  past  philosophy  that 
environmental  efforts  were  not  part  of  mainstream, 
conventional  programs.  The  lack  of  examples  made  output 
measure  development  especially  difficult. 


3.  Some  of  the  measures  developed  by  the  author  and 
validated  by  Air  Force  experts  are  more  meaningful  than 
others.  For  example.  Total  Documents  Reviewed  (II)  and 
Total  Reports  Prepared  (12)  have  relatively  less  validity 
since  organizations  such  as  the  EPF  normally  do  not  track 
them  and  variability  in  document  type  and  length  can  be 
great.  On  the  other  hand,  documenting  and  tracking  some 
paperwork  are  important  duties  of  environmental  organization 
personnel.  Environmental  Impact  Analysis  Process  documenta¬ 
tion  (06)  is  a  critical  aspect  of  base  work  and  personnel 
should  be  aware  of  this  output.  Managers  may  want  to  weight 
measures  if  they  feel  this  type  of  disparity  in  the 
importance  of  measures  exists. 

4.  Output  measures  are  more  difficult  to  develop  than 
input  measures.  For  example,  the  output  measure.  Responses 
to  Environmental  Incidents  (04),  was  intended  to  show  how 
much  the  base  EPF  could  respond  with  given  resources. 
Conversely,  a  base  with  good  training  for  personnel  handling 
hazardous  materials  and  good  spill  prevention  plans  should 
need  to  respond  to  incidents  relatively  less  often. 

Responses  (04)  may  show  EPF  efficiency,  but  the  measure  may 
be  in  direct  opposition  to  effectiveness  measures,  such  as 
Oil  Spills  (015)  which  measures  effectiveness  in  preventing 
incidents.  Evaluating  efficiency  (e.g.,  response)  and 
effectiveness  (e.g.,  spill  prevention)  separately  is  one  way 
to  avoid  this  problem 


Objective  2.  The  second  objective  was  to  collect  data 
on  the  measures  established  for  the  EPFs.  Using  a  survey 
sent  out  by  the  Air  Force  Engineering  and  Services  Center 
(AFESC)  under  Project  IMAGE,  data  from  CONUS  installations 
was  collected  to  use  in  testing  the  analysis  model.  The 
following  are  conclusions  regarding  Objective  2: 

1.  The  surveys  are  rich  in  information  about 
environmental  planning.  In  addition  to  the  data  used  for 
performance  analysis,  otner  information  is  available  to 
evaluate  program  trends  and  regional  differences. 

2.  Data  collected  for  some  of  the  survey  elements 
showed  extreme  variations  in  values.  As  mentioned  in 
previous  chapters,  measurement  error  probably  occurred  in 
some  data  elements,  such  as  service  contracts  (15).  Other 
variations  occurred  because  of  differences  in  base  programs 
or  state  and  regional  requirements.  As  pointed  out  by  some 
respondents,  some  states  require  each  air  pollution  source 
to  be  permitted  separately,  while  other  states  allow  all 
sources  to  be  covered  under  just  a  few  permits.  Accounting 
for  these  variations  during  data  collection  would  be 
difficult,  but  may  be  necessary  to  ensure  proper  EPF 
comparison. 

Unfortunately  much  of  the  disparity  and  error  may  have 
been  caused  by  shortcomings  in  record  keeping  practices. 

Some  respondents  had  difficulty  making  estimates  about 
environmental  documentation  data  which  could  be  requested  at 
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anytime  by  regulatory  agencies  or  by  individuals  under  the 
Freedom  of  Information  Act.  Other  evidence  of  poor  records 
was  that  the  Number  of  Inspections  (03)  often  did  not  meet 
the  minimum  set  by  regulatory  agencies  for  various  types  of 
pollution  control  facilities.  These  data  base  gaps  tend  to 
confirm  Nash's  contention  (31:159),  presented  in  Chapter  3, 
that  performance  information  is  seldom  available  in  usable 
form.  Even  though  EPF  personnel  could  answer  the  surveys, 
they  do  not  collect  all  of  the  data  on  a  regular  basis. 

Objective  3.  The  third  objective  was  to  analyze  the 
data  using  an  analytical  method  considered  appropriate  for 
EPFs.  Constrained  Facet  Analysis  (CFA)  was  selected  for  its 
capability  to  evaluate  multiple  inputs  and  outputs  of 
varying  units  of  measure.  CFA  was  also  preferable  to  the 
similar  Data  Envelopment  Analysis  since  it  could  evaluate 
outlier  units  with  dissimilar  mixes  of  inputs  and  outputs. 
According  to  the  literature  review,  CFA  has  established 
credibility  in  some  public  agencies  (primarily  schools)  and 
has  been  shown  to  have  potential  applicability  in  the  Air 
Force.  Even  so,  the  author  is  not  aware  of  any  attempt  by 
the  Air  Force  to  examine  CFA  capabilities  in  comprehensive 
studies,  including  actual  field  tests. 

Objective  The  intent  of  the  fourth  objective  was 
to  analyze  the  test  results  and  evaluate  the  success  of  the 
selected  measures  and  model.  CFA  was  able  to  analyze  the 
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data  and  provide  results  which  rated  the  efficiency  of  34 
EPFs.  Most  of  the  test  runs  were  successful  in  differen¬ 
tiating  the  EPFs  on  the  basis  of  either  their  efficiency  in 
office  work  or  effectiveness  in  compliance.  Some  test  runs 
were  unsuccessful  because  of  model  and  measure  characteris¬ 
tics  discussed  below. 

1.  Limitations  of  the  Productivity  Assessment  Support 
System  computer  program  and  the  CFA  model  restrict  their 
practical  use.  Research  by  other  Air  Force  Institute  of 
Technology  (AFIT)  graduate  students  revealed  similar 
shortcomings. 

a.  The  computer  program  was  somewhat  "user 
unfriendly"  and  a  few  logic  errors  exist.  Considering  the 
amount  of  manipulation  required,  data  entry  would  be 
troublesome  for  managers,  probably  requiring  a  technician. 
Furthermore,  the  microcomputer  data  processing  and  reporting 
operations  were  time-consuming  (the  author  took  over  60 
hours  during  this  phase).  Finally,  some  of  the  reports  are 
difficult  to  interpret  without  a  working  knowledge  of  CFA 
theory  and  operation. 

b.  CFA  appeared  to  have  difficulty  handling 
extreme  value  ranges  of  variables.  Byers  and  Waylett 
noticed  that  if  the  range  was  too  large  the  linear 
programming  matrix  often  failed  to  produce  meaningful 
results  (11:118).  Variables  with  value  ranges  of  0  to 
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999/999  were  being  evaluated  in  some  tests.  The  Test  1  - 
Overall  analysis,  which  gave  efficient  ratings  (1.0)  to  all 
34  EPFs,  may  have  been  influenced  by  the  wide  range  of 
variable  values.  Successful  tests  not  only  had  fewer 
variables  being  compared,  but  had  variables  with  similar 
value  ranges. 

c.  Another  limitation  was  that  the  model  would 
not  process  zero  input  values,  so  they  were  entered  as 
999,999.  For  the  Contract  Service  Costs  (15)  variable,  this 
meant  that  both  $0.00  and  $1,000,000.00  were  entered  as 
$999,999.00. 

2.  Late  survey  returns  and  difficulties  in  computer 
processing  restricted  the  number  of  tests  accomplished. 
During  the  various  testing  stages  the  need  for  additional 
testing  became  apparent. 

a.  For  proper  data  representation.,  reciprocal 
values  were  only  used  for  selected  variables.  Using 
reciprocals  is  a  legitimate  research  technique,  but  has  the 
limitation  of  not  always  maintaining  linear  relationships  in 
the  data  which  may  have  existed  prior  to  conversion.  The 
Compliance  variable  (08,10,12,14,17)  was  a  reciprocal  value 
for  the  number  of  air,  water,  and  solid  waste  compliance 
problems.  These  compliance  problems  were  also  used  as 
individual  output  variables  in  nonreciprocal  form.  A 
comparison  of  these  variables  as  X  versus  1/X  may  have  shed 
more  light  on  the  appropriateness  of  using  reciprocal 
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b.  A  test  run  evaluating  all  organizational 
measures  was  never  completed.  The  number  of  observations 
was  not  sufficient  to  run  against  the  17  variables  in  Test 
1  -  Overall.  More  observations  could  come  from  having  more 
base  responses  or  evaluating  EPFs  in  longitudinal  studies 
(i.e.,  over  several  time  periods).  Fisher  and  Kellog  noted 
that  measuring  total  unit  productivity,  not  just  component 
parts,  was  necessary  to  identify  all  inefficiencies  within 
an  organization  (23:98;  29:142). 

c.  Because  of  the  restriction  on  the  number  of 
variables,  values  for  six  manhours  variables  (01,2,3,4,5,6) 
were  not  entered.  Thus,  manhours  inputs  (12)  were  never 
compared  to  number  of  manhours  output  (01,2,3,4,5,6).  The 
results  of  comparing  identical  measurement  units,  manhours, 
would  have  given  further  insight  into  CFA’s  capabilities. 

Objective  5^.  The  final  objective  was  to  discuss  the 
feasibility  of  CFA  application  given  the  analysis  results. 
Chapter  VI  discussed  in  detail  the  potential  CFA  applica¬ 
tions  and  implementation  strategies.  The  main  conclusion  is 
that  CFA  appears  to  be  useful  for  Air  Staff,  Major  Command, 
and  EPF  managers  to  use  in  making  resource  allocation  and 
program  emphasis  decisions.  As  Nash  stated  (31:174),  just 
the  act  of  measuring  helps  to  clarify  goals  and  assess 
priorities. 

1.  A  comparison  of  EPF  rankings  for  process  and 
results  tests  showed  no  strong  correlation  between  EPF 
efficiency  and  effectiveness.  Of  course,  the  poor 
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correlation  does  not  disprove  the  proposition  that  a  base 
will  improve  compliance  as  it  increases  efficiency.  The 
lack  of  correlation  is  understandable  considering  EPFs 
frequently  are  forced  to  abandon  efficient  courses  of  action 
in  order  to  react  to  a  new  onslaught  of  regulations. 

2.  Of  more  significance  to  managers  is  the  potential 
of  using  either  CFA  reports  or  model  bases  to  improve  EPFs. 
CFA  reports  can  indicate  which  measures  are  contributing 
poorly  to  EPF  efficiency  or  effectiveness.  Managers  can 
plan  operational  changes  and  evaluate  their  effect  using 
computer  program  subroutines  which  predict  outcomes.  Other 
planning  assistance  comes  from  examining  efficient  units  and 
comparing  them  to  inefficient  EPFs  or  program  areas.  The 
frontier  reports  indicate  which  EPFs  are  similar  in  input 
and  output  mixes  so  the  comparison  is  more  significant. 

3.  Actual  use  of  CFA  tools  by  EPF  managers  would  tend 
to  be  base  specific.  Since  base  environmental  programs  are 
interdisciplinary  in  nature  and  directed  by  Environmental 
Protection  Committees,  many  EPFs  tend  to  emphasize  programs 
and  approach  compliance  differently.  And  EPFs  take 
different  operational  approaches  to  state/regional  political 
and  regulatory  considerations  which  reduces  the  chances  for 
a  consistent  pattern.  Using  similar  techniques  to 
manufacture  or  repair  items,  production  oriented 
organizations  tend  to  have  more  operational  consistency  from 
base  to  base. 


Recommendations 


This  research  indicates  that  EPFs  can  successfully  be 
characterized  in  terras  of  input  and  output  measures  and 
analyzed  using  a  quantitative  model.  The  following 
recommendations  are  made  with  this  in  mind. 

1.  Continue  to  study  performance  factors  of  private 
and  public  environmental  organizations.  The  continual 
growth  in  the  organizational  significance  of  environmental 
functions  warrants  academic  institutions,  such  as  AFIT, 
pursuing  this  research.  A  more  in-depth  review,  possibly 
including  primary  data  sources,  may  find  research  which  can 
more  directly  contribute  to  the  characterization  of 
environmental  organizations  and  their  measures. 

2.  Re-evaluate  the  EPF  measures  and  their  variable 
counterparts,  and  re-validate  that  CFA  is  appropriate  for 
EPFs.  More  of  the  EPF  operations,  such  as  the  Installation 
Restoration  Program,  may  need  to  be  included.  However,  the 
evaluation  should  consider  dropping  the  less  important 
measures  and  consolidating  other  measures  or  variables. 

3.  Re-program  the  Productivity  Assessment  Support 
System  so  it  is  more  user  friendly  and  practical  for 
managers.  The  users  manual  (8)  should  be  expanded  to 
include  more  information  on  the  interpretation  of  results. 
Also,  re-configure  the  program  so  that  more  than  20 
variables  can  be  evaluated  at  once  (as  allowed  by  CFA 
theory)  and  so  there  is  more  flexibility  in  choosing 
variable  combinations. 
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4.  Continue  to  collect  and  analyze  data  to  further 
characterize  the  EPFs.  Recommend  that  this  survey 
information,  and  other  data  collected  under  Project  IMAGE, 
be  retained  by  the  Engineering  and  Services  Center  so  that 
more  research  can  be  attempted.  For  example,  correlation 
studies  on  EPF  performance  in  relation  to  location  could 
show  state  or  EPA  Region  trends.  Ratios  of  personnel 
assigned  to  personnel  authorized  compared  to  efficiency  or 
effectiveness  could  indicate  effects  of  manning  levels  on 
EPF  performance. 

5.  Measure  Air  Force  EPF  efficiency  and  effectiveness 
on  a  regular  basis  using  a  combination  of  qualitative  and 
quantitative  means.  Quite  similarly,  Byers  and  Waylett 
advocated  an  "integrated  automated  management  information 
system"  (11:117).  The  Defense  Environmental  Status  Report 
could  serve  as  one  qualitative  evaluation  method  and  CFA 
could  serve  as  the  quantitative  technique.  Proper  measure¬ 
ment  is  important  to  demonstrate  Air  Force  compliance  and  is 
critical  to  justify  increases  in  funding  and  manpower. 
Project  IMAGE  efforts  and  CFA  analyses  (including  AFIT 
theses)  should  be  integrated  with  manpower  studies  to 
improve  Air  Force  EPF  resources. 

6.  As  an  adjunct  to  the  previous  suggestion, 
institute  a  better  environmental  record  keeping  system.  The 
Work  Information  Management  System  (WIMS)  appears  to  be  an 
ideal  system  for  accomplishing  this.  Headquarters  Air  Force 


and  Major  Commands  should  develop  criteria  for  the  types  of 
data  to  be  collected  and  evaluated.  HQ  USAF/LEEV  should 
strengthen  Air  Force  accountability  procedures  for 
compliance  data.  HQ  USAF/LEEV  should  also  work  with  the 
WIMS  monitor  to  improve  methods  for  environmental  analysis 
documentation  under  existing  work  tracking  procedures.  And 
separate  record  keeping  categories  for  the  environmental 
media  (air,  water,  and  solid  waste)  need  major  expansions  to 
accommodate  information  for  audit  and  analysis  purposes. 

7.  Implement  some  performance  evaluation  model, 
preferrably  CFA,  Air  Force  wide.  Most  of  the  advantages  and 
disadvantages  of  CFA  have  been  well  documented,  so  further 
study  of  the  model  for  individual  decision  making  units 
appears  to  be  of  marginal  value.  The  next  step  should  be  a 
feasibility  study  to  demonstrate  CFA's  capabilities  in 
either  Civil  Engineering  or  total  Air  Force  settings.  HQ 
AFESC/DEM  should  take  the  lead  for  Civil  Engineering  in 
working  with  other  Air  Force  functional  areas  in  pursuing 
the  idea.  HQ  USAF/LEEV  should  validate  the  need  for  the 
feasibility  study  and,  if  appropriate,  provide  support  for 
acquiring  CFA  because  of  its  importance  in  evaluating  EPF 
performance . 

The  above  recommendation  is  made  with  two  reserva¬ 
tions.  The  first  is  that  the  author  feels  CFA  implementa¬ 
tion  could  not  be  justified  solely  for  EPF  use.  The  second 
reservation  is  that  implementation  is  attractive  only  if  CFA 


can  be  placed  on  existing  systems.  The  Air  Force  should 
acquire  the  rights  to  modify  the  computer  software  so  that 
it  can  be  placed  on  Air  Force  systems  and  adapted  to  our 
requirements.  WIMS  would  be  the  ideal  system  for  Civil 
Engineering/  especially  if  the  computer  model  could  have 
direct  access  to  an  augmented  environmental  data  base. 


Appendix  A:  Review  of  Productivity  Analysis  Methods 
(15:53-62) 

Regression 

Ordinary  least  squares  regressions  of  the  single¬ 
output,  multiple-input  variety  having  both  positive  and 
negative  error  terms  produce  curves  of  average  relationship. 
These  curves  do  not  represent  frontiers,  which  by  definition 
are  based  on  extremal  relations.  Actual  output  values  lie 
above  and  below  the  regression  curve,  and  the  outputs  of 
efficient  units  are  not  necessarily  greater  than  their 
corresponding  regression  estimates.  With  stochastic  models, 
additional  information  is  gained  by  decomposing  residuals, 
but  for  frontier  estimation  in  the  type  of  problems  which 
have  been  addressed  by  DEA,  average  estimates  are 
uninformative.  Furthermore,  in  some  cases  the  size  and 
direction  of  the  residuals  may  appear  to  have  little  or  no 
bearing  on  the  efficiency  measure  (distance  from  the 
frontier ) . 

A  major  difficulty  arises  in  the  multiple  output  case 
when  least  squares  regression  analysis  is  performed  on  each 
output  separately.  The  other  outputs  excluded  from  the 
analysis  have  an  implicit  impact  since  they  may  rely  on 
(compete  for)  the  same  resources.  Each  regression  equation 
might  be  able  to  predict  adequately  an  expected  level  of  a 
single  output  for  an  organization,  assuming  this  organiza¬ 
tion  could  experience  any  of  the  random  fluctuations  or 
inefficiencies  of  the  industry  (all  firms)  and  recognizing 
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that  the  influence  of  other  outputs  are  implicitly  taken 
into  account  by  the  deviations  from  the  regression  line 
(residuals).  But  these  equations  cannot  predict  the 
expected  output  of  an  organization  whose  variations  and/or 
inefficiencies  are  significantly  affected  by  the  given 
technology  and  policies  of  the  firm  which  are  not  random. 
Magnitudes  of  actual  outputs  of  an  organization  are 
influenced  by  both  local  and  corporate  policy  which  may 
prevent  the  true  expected  output  values  of  the  organization 
from  conforming  to  the  corresponding  regression  estimates. 
Furthermore,  there  might  be  little  or  no  correlation  between 
the  relative  magnitude  of  actual  organizational  outputs  and 
the  relative  magnitudes  of  their  regression  estimates,  yet 
relative  magnitudes  have  an  important  effect  on  the 
establishment  of  frontiers  and  neighborhoods  of  comparison 
in  multiple  output  situations. 

If  a  linear  least  squares  regression  equationy  with  all 
of  its  assumptions,  is  accepted  as  a  proper  representation 
of  organizational  productivity,  then  according  to  Sherman 
the  "relationships  estimated  by  regression  techniques 
reflect  (approximately)  efficient  input-output  relation¬ 
ships."  The  rate  of  technical  substitution  of  any  two 
inputs — the  ratio  of  the  regression  coefficients  for  those 
inputs — is  assumed  constant  for  all  organizations. 

Moreover,  the  rate  at  which  an  input  is  transformed  into  an 
output  is  assumed  to  be  the  same  for  every  organization. 
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Under  these  assumptions,  the  average  output  of  an 
organization  is  not  expected  to  increase  unless  one  or  more 
of  the  inputs  increase;  and,  if  any  input  is  reduced,  then 
another  input  must  increase  or  the  expected  output  will  be 
reduced.  These  relationships  are  those  that  one  would 
expect  to  find  in  organizations  having  efficient  productive 
capability.  In  inefficient  organizations,  either  the  same 
output  can  be  achieved  with  reduced  inputs  or  greater  output 
can  be  achieved  with  reduced  inputs  or  greater  output  can  be 
obtained  with  the  same  resources. 

Perhaps  Sherman  used  the  term  "approximately"  in 
allowing  for  the  random  output  variability  represented  by 
the  regression  residual.  This  variability  is  assumed  to  be 
caused  by  reasonable,  efficient  adjustments  of  output 
levels  in  response  to  random  shocks  in  production  or  random 
market  fluctuations.  However,  if  organizations  are 
operating  under  different  technologies,  the  variances  in 
outputs  caused  by  differences  in  technical  efficiency  would 
be  subsumed  by  this  residual  term.  These  variances  are  not 
random.  Two  organizations  having  precisely  the  same  inputs 
but  different  levels  of  technical  efficiency  would  also  have 
two  different  expected  output  levels,  and  the  difference 
would  be  accounted  for  in  the  residual  term.  Least  squares 
would  consider  efficient  and  inefficient  organizations 
simultaneously  and  the  best  fit  would  be  influenced  by  both 
types  of  behavior,  including  the  case  where  residuals  are 


forced  to  lie  below  the  frontier.  Under  such  conditions,  a 
single  regression  equation  would  misrepresent  the  productive 
capaoility  and  efficiency  of  the  units. 

One  would  expect  that  removal  of  the  subsumed 
difference  in  technical  efficiency  from  the  residual  term 
would  produce  a  regression  equation  which  explains  more  of 
the  variation  in  output  (higher  R2).  Sherman  tested  this 
hypothesis  with  a  simulation;  and  Bessent,  Bessent  and  Clark 
were  able  to  support  his  findings  by  using  DEA  to  identify 
the  efficient  organizations  in  a  sample  (n-216),  then 
applying  least  squares  regressions  to  the  efficient  units 
only,  and  comparing  these  regression  results  with  the  ones 
obtained  when  all  units  were  considered. 

The  R2  value  increased  when  only  efficient  units  were 
used  in  the  regression.  There  were  also  modest  gains  in  the 
significance  levels  of  regression  coefficients.  These 
improvements  were  achieved  despite  the  reduction  in  residual 
degrees  of  freedom. 

If  there  exist  an  adequate  number  of  efficient  units  in 
the  data  set,  the  above  results  suggest  that  one  should 
perform  regression  (linear  or  nonlinear)  on  only  the 
efficient  units  to  obtain  the  best  regression  equation,  one 
which  comes  closer  to  representing  a  frontier  and  which 
explains  a  larger  portion  of  output  variability. 


But,  it  appears  at  this  point  that  current  regression 
analysis  techniques  are  largely  inappropriate  when  estab¬ 
lishing  frontiers  for  nonprofit  organizations  which  do  not 
have  highly  mechanistic,  scientifically  engineered  produc¬ 
tion  technologies.  DEA,  on  the  other  hand,  provides  a 
useful  representation  of  an  attainable  production  frontier, 
provides  pertinent  information  about  organizational 
efficiency,  and  is  not  subject  to  the  errors  and  misrepre¬ 
sentations  which  can  result  if  the  regression  assumptions 
are  violated  or  if  the  from  of  the  production  function  is 
misspecif ied . 

Furthermore,  DEA  takes  all  outputs  and  inputs  into 
account  simultaneously  including  differences  in  input/output 
mixes  and  tradeoffs  among  factors.  It  indicates  which 
organizations  are  on  the  efficiency  frontier,  establishes  a 
piece-wise  linear  approximation  of  the  frontier  surface 
using  efficient  units,  and  assigns  an  efficiency  measure 
based  on  how  far  the  unit  is  from  a  frontier  point  directly 
between  the  unit  and  the  origin,  a  point  for  which  input  and 
output  values  are  linear  combinations  of  the  observations 
from  an  efficient  set  of  "neighborhood"  organizations. 
Evaluations  of  frontier  points,  neighborhoods,  and 
efficiency  measures  for  individual  units  are  all  readily 
accessible  through  DEA  but  are  hidden  from  explicit 
examination  in  the  regression  analysis. 


Although  DEA  appears  to  be  the  best  alternative  for 
analysis  in  the  public  sector,  further  research  is  needed  to 
determine  how  the  models  of  DEA  and  statistical  econometrics 


can  be  used  in  conjunction  with  one  another  to  improve 
frontier  estimation  and  analysis.  It  is  likely  that  a  more 
thorough  frontier  analysis  will  be  achieved  by  the  use  of  a 
number  of  different  but  related  models. 

Ratio  Analysis 

Ratio  Analysis  is  not  a  method  of  frontier  estimation, 
but  it  is  relevant  to  this  discussion  because  of  its 
frequent  use  as  an  ex  post  facto  evaluation  tool  in 
analyzing  multiple  input,  multiple  output  relations.  Users 
of  this  method  examine  multiple  measures  in  the  form  of 
ratios  in  an  attempt  to  compare  the  performance  of  similar 
organizations;  each  ratio  typically  being  a  single  output 
measure  divided  by  a  single  input  measure.  Like  DEA,  ratio 
analysis  is  used  when  the  production  process  in  unknown  or 
difficult  to  model. 

Unlike  DEA,  ratio  analyses  do  not  make  use  of 
mathematical  models  to  organize  or  assimilate  ratios  into  a 
single  aggregate  measure  of  efficiency;  i.e.,  they  do  not 
simultaneously  take  into  account  interactions  over  the  full 
range  of  inputs  and  outputs.  As  a  result,  the  performances 
of  organizations  are  difficult  to  compare  using  this  method 
particularly  when  organizations  rank  comparatively  high  on 
some  measures  and  low  on  otners. 


This  difficulty  can  be  illustrated  by  the  following 
simple  example.  Consider  the  two  organizations  in  Table 
2.1. 


Ratio  2  (=Output/Input  2)  0.25  0.4 

Note,  Ratio  1  of  organization  A  is  larger  than  Ratio  1  of  B, 
and  the  situation  is  reversed  for  Ratio  2.  The  relative 


performance  of  organizations  A  and  B  cannot  be  determined  by 
examination  of  these  ratios  unless  the  relative  importance 
(weight)  of  each  ratio  is  specified.  Furthermore,  as  the 


number  of  inputs  and  outputs  increases,  the  problems  of 
weighting  and  assimilation  grow  raul tipi icatively . 

Lewin,  Morey,  and  Cook  examined  this  problem  in  an 


evaluation  of  judicial  districts.  They  ranked  each  of  ten 


output-to-input  ratios  (2  outputs  x  5  inputs)  and  displayed 
the  number  of  times  that  districts  were  ranked  in  the  upper 
and  lower  quartiles  over  the  ten  ratio  measures.  Several 
districts  were  noted  to  have  ratios  in  both  quartiles. 

Under  these  circumstances,  it  would  be  very  difficult  to 


find  a  simple  rule  to  distinguish  efficient  districts  from 
inefficient  ones  without  making  subjective  judgments  about 
the  relative  importance  of  each  ratio. 

Another  related  difficulty  stems  from  the  fact  that 
single  ratios  provide  only  partial,  incomplete  measures  of 
multiple  input-output  relations,  a  condition  which  often 
leads  to  incorrect  judgments  of  performance.  In  actual 
practice,  partial  measures  such  as  "units  produced  per 
manhour"  are  used  as  measures  of  performance  without  regard 
to  other  inputs  such  as  supplies,  fuel,  equipment,  etc.  The 
data  in  Table  1  can  be  used  to  illustrate  the  risk  in  this 
practice,  if  one  were  to  compare  units  A  and  B  based  on 
Ratio  1  alone,  unit  B  would  appear  to  be  a  better  performer 
than  A  (0.4  >  0.25),  but  the  reverse  would  be  true  if  Ratio 
2  were  considered  alone. 

Sherman  and  Bessent,  Bessent  and  Clark  used  DEA  as  a 
vehicle  for  examining  the  risks  of  partial  measurements. 

Each  experiment  used  a  set  of  hypothetical  organizations 
whose  inefficiencies  were  known  and  detectable  by  DEA.  DEA 
efficiency  evaluations  were  performed  on  the  sets  with  all 
inputs  and  outputs  included.  Other  evaluations  were 
performed  with  one  input  or  output  omitted.  The  results 
indicated  that  partial  measures  can  cause  misclassif ications 
of  efficiency;  i.e.,  organizations  might  be  incorrectly 
labeled  efficient  or  inefficient,  or  the  magnitudes  and 
causes  of  the  inefficiencies  might  be  misspecif ied .  In 
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general,  the  omission  of  relevant  inputs  or  outputs  during 
frontier  evaluations  may  cause  distorted  neighborhoods  of 
comparison,  erroneous  slack  conditions  or  measurements 
relative  to  the  wrong  frontier  facet. 

Despite  the  above  shortcomings,  ratios  do  have  the 
advantage  of  being  familiar  to  managers  and  simple  to 
understand.  But  this  advantage  is  outweighed  by  the  risk  of 
obtaining  misleading  results  unless  ratio  analysis  is  used 
in  conjunction  with  methods  of  frontier  estimation  like  DEA 
which  are  able  to  take  all  inputs  and  outputs  into  account 
simultaneously. 
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Appendix  B:  Environmental  Quality  Program  Goals 


Clean  Air  Goals 

Comply  with  the  Clean  Air  Act  oy: 

o  Identifying  and  correcting  all  air  pollution 
deficiencies. 

o  Obtaining  required  permits,  variances,  and 
compliance  agreements. 

o  Completing  required  transportation  control  plans. 

o  Implementing  vehicle  inspection  and  maintenance 
( I&M)  program  where  required. 

o  Completing  required  air  episode  plans. 

Clean  Water  Goals 

Comply  with  the  Clean  Water  Act  by: 

o  Identifying  and  correcting  all  water  pollution 
deficiencies. 

o  Obtaining  required  permits  and  compliance 
agreements . 

o  Using  municipal  or  regional  treatment  facilities 
when  economically  feasible  or  required  under 
approved  plans. 

o  Developing  and  implementing  oil  Spill  Prevention, 
Control,  and  Countermeasures  (SPCC)  plans  and 
programming  required  projects. 

o  Eliminating  oil  and  hazardous  substance  spills  from 
installations  and  ships. 

Solid  and  Hazardous  Waste  Goals 

Comply  with  the  Resource  Conservation  and  Recovery  Act 

(RCRA),  and  the  Toxic  Substances  Control  Act  (TSCA) 

by : 

o  Complying  with  the  regulations  for  handling, 

storage,  transportation,  and  disposal  of  hazardous 
waste . 


o  Complying  with  the  regulations  for  the  handling, 
storage,  marking,  and  disposal  of  PCBs  and  PCB 
items. 

o  Maximizing  the  utilization  of  used  POL  products, 
particularly  the  sale  of  lubricating  oil  for  re¬ 
refining.  Use  sale  proceeds  for  pollution 
abatement;  energy  conservation;  occupational  safety 
and  health;  and  morale  and  welfare  projects. 

o  Implementing  source  separation/recycling  programs 
at  all  DOD  installations  where  economically 
feasible. 

o  Minimize  waste  generation  through  process  modifi¬ 
cation,  recycle,  reuse,  materials  substitution  or 
other  methods  where  economically  feasible. 

o  Developing  and/or  participating  with  municipalities 
in  resource  recovery  facilities  programs  for 
Standard  Metropolitan  Statistical  Areas  (SMSA) 
where  feasible. 

Installation  Restoration  Goals 

Implement  an  Installation  Restoration  (Superfund) 

program  in  accordance  with  Executive  Order  12315  and 

DEQPPM  81-5  by: 

o  Cooperating  with  state  and  EPA  Regional  officials 
by  providing  timely  information  on  IR  program 
status,  to  include  copies  of  completed  Phase  I  and 
Phase  II  reports,  and  providing  other  significant 
monitoring  data  as  requested. 

o  Identifying  sites  for  Phase  II  (confirmation 
studies  or  surveys)  on  a  continuing  basis. 

o  Programming  for  Phase  IV  (remedial  action)  as 
requirements  are  identified.  Complete  remedial 
actions. 

Safe  Dr  inking  Water  Goals 

Provide  safe  drinking  water  at  DOD  activities  and 

comply  with  the  Safe  Drinking  Water  Act  (SDWA)  by: 

o  Assuring  that  all  DOD-owned  or  operated  public 
water  supplies  are  constructed,  operated,  and 
maintained  in  compliance  with  the  SDWA  and 
implementing  EPA  regulations. 
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o  Performing  analyses  for  chemical,  physical, 
biological,  and  radiological  contaminants  as 
required  by  EPA  or  state  drinking  water  standards. 

o  Assuring  that  drinking  water  quality  on  DOD 

installations  complies  with  standards  established 
by  the  SDWA. 

o  Assuring  that  water  treatment  personnel  are  properly 
trained. 

Environmental  Auditing  Management  Goals 

o  Improve  compliance. 

o  Provide  assurance  to  management  that  its  activities 
do  not  contribute  to  environmental  problems  which 
would  expose  the  government  to  large  future 
financial  liabilities  or  significantly  degrade  the 
environment. 


o  Provide  a  means  of  achieving,  maintaining  and 
monitoring  compliance  on  a  continuous  basis. 


Appendix  C:  Environmental  and  Contract  Planning  Work 
Center  Description 

Natural  Resource  Planning  -  prepares,  manages,  coordinates, 
implements  and  maintains  the  following  programs  and 
supporting  plans: 

Land  Management  Program/Plan 

Landscape  development 
Golf  course  management 
UrDan  forestry 

Documents  for  prime  and  unique  farmland 
Wetlands  and  flood  plains 
Wild  and  oceanic  rivers 

Graying  and  Cropland  Management  Program  Plan  (includes 
applicable  outleases) 

Forest  Management  Program/Plan 

Endangered  species  (including  biological  assessments) 

Outdoor  Recreation  Program/Plan 

Natural  Resource  Planning 

Off  road  vehicles 
Picnic  area  management 
Bird  watching 
Nature  trails 
Photography 
Hiking  trails 
National  historic  trails 
Natural  areas 

Bird  Aircraft  Strike  Hazard  (BASH)  Program/Plan 
Nuisance  birds 

Historic  Preservation  and  Archeology  Program/Plan 
Cultural  resources  management  plan 
Pest  Management  Program/Plan 

Nuisance  animals,  rodents 

NOTES:  Other  natural  resource  plans  are  or  may  be  required 
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Management  of  preceeding  plans  includes  performing  environ¬ 
mental  assessments,  cooperative  agreements,  memorandum  of 
understanding  (MOUs)  as  well  as  preparing,  coordinating  and 
gaining  approval  of  reports  as  applicable. 

Additional  Areas  of  Responsibility 

Is  the  single  point  of  contact  with  local,  state  and 
federal  agencies  for  natural  resources  activities. 

Reviews  contract  drawings  and  specifications  for 
natural  resources  impact. 

Prepares/coordinates  and  reviews,  in-house  and 
contract  environmental  impact  and  assessment  process 
( EIAP )  documents. 

Prepares  submittal  for  AF  and  DOD  natural  resource 
award  programs 

Prepares  input  to  public  affairs  for  press  release 

Advises  base  executive  staff  (including  tenant  units) 
on  all  natural  resource  management  programs  and  plans 

Overall  manages  the  development,  coordination,  implementa¬ 
tion  and  maintenance  of  plans  and  programs  related  to  the 
installation’s  base  comprehensive  plan  (BCP)  to  include  all 
component  elements  indicated  in  AFR  86  — <=  ( 26  Dec  84). 

BCP  Program  Management 

In-house  planning 


Initiatives/Promotes 
functions : 

Planning 
Planning 
Planning 
Planning 
Host  and 

IPF 

Plan  maintenance 


Linkages  to  related 


future  EIAP 

and  real  estate/space  survey 
related  studies 
and  programmers 
tenant  units 


Presentation/briefings 

Originates/organizes  'at's  and  other  studies 
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Contracts 


Prepare  procurement  package  "SOW" 
AE  selection 
Manage  contract 
Implements/ in teg rate 


IICEP 


Conforms,  coordinates  witn  off-base  planning 
organizations 

MOUs 

Encroachment 


AICUZ 


Prepares  data  package 
Reviews/revises  products  from  AFESC 
Coordinates  with  MAJCOM,  HQ  USAF 
Release  study/public  review 

Land  use  Planning 


Prepares  and  maintains  land  use  plan 

Ensure  sitings  are  compatible  to  land  use  plan 

Prepare  area  development  plans 

Manages  F-13  agenda/activities 

Ensures  1391s  are  compatiDle 

Provides  assistance  to  functional  users 

Coordinates  input  to  mission  planner 

Ensure  compatibility  with  airfield/airspace 

Ensure  adequate  land  buffer  for  development 

Transportation  Planning 

Prepare  master  transportation  plan 
Plan  traffic  engineer  studies 
Interact  with  MTMC 

Reviews  projects  for  traffic  impact 
Range  Planning 

Prepares/maintains  range  plan 
Establish  range  selection  criteria 
Coordinates  with  operational  functions 
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Environmental  Plannin 


Preparation  and  coordination  of  the  environmental 
impact  analysis  process 

Prepares  for  participation  in  EPC  and  supporting 
working  groups 

Acts  as  base  liaison  with  federal,  state,  local 
and  public  agencies 

Maintains  and  prepares  environmental  records  and 
reports  which  includes: 

DRM,  DESR,  1383,  Spill  Reports,  Inspection 
Reports  ( PCB ,  Hazardous  Waste) 

Submittals  for  incentive  awards  programs 

Management  of  environmental  compliance  assessment  and 
management  program 

Reviews,  coordinates  O&M,  MCP  project  documents 

Provide  technical  assistance/training  to  EPC,  AF 
community,  other  installation  organization 

Coordinates  in  long  range  planning 

Monitor  implementation  of  AICUZ  and  noise  program 

Operates  environmental  complaints/suggesting  program 

Program  for  future  environmental  requirements 

Maintains  currency  with  all  environmental  regulation 
charges 

PCB 

Develop/iraplement/raonitor  PCB  inventory  control 
program 

Manage/inspect  PCB  storage  facilities 
Coordinates  PCB  disposal 

Programming  for  removal  of  in  service  PCB  transtore- 
raent 

Prepare  required  reports 
Hazardous  Waste 


Manages  Phase  I 


Evaluates  potential  for  new  sites 
Maintains/updates  data  base 

Coordinates  actions  with  federal  state/  local 
governments 

Coordinate  Phase  II 

Review  SOW  and  Phase  II  draft  and  final  reports 
Provides  support  to  Phase  II  contractor 

Manages/ implements/maintains  Phase  IV 

Develops  SOW  for  and  monitors/oversees  RAPS 

Makes  remedial  action  alternative  selection  and 
defends  selections  to  AFIRM  committee 

Develops  design  for  remedial  action 

Monitors  remedial  action  construction 

Coordinate  action  with  federal,  state  and  local 
agencies 

Develops  and  maintains  long  term  monitoring 
program  and  O&M  requirements  needed  as  result 
of  IRP  action 

Hazardous  Waste  Reduction 

Identif ies/reviews/impleraents  reduction  strategies 
Hazardous  Waste  Management 

Obtains/maintains  R&D  permits 

Develops  &  implements  &  maintains  H  W  Management  plan 
Identifies/maintains  waste  stream  inventory 
Develops/implements  training  program  for  H  W  Handlers 
Maintains  currency  with  regulatory  changes 
Documents  and  maintains  records 
Inspects  accumulation  point  and  other  TSD 
Manages  H  W  storage  areas 


Annually  surveys  H  W  disposeal  areas 
Prepares/manages  H  W  Disposal 
Participates  in  emergency  response 
Coordinates  inspection  by  outside  agencies 
Develops  and  implements  correct  action 

Lust 

Inventory  existing  and  decommissioned  facilities 

Develops/ iraplement/maintain  product  inventory  control 
testing  program 

Plan  and  program  for  current/future  remedial  action 
Ensure  integrity  of  physical  plant  maintenance  plan 
Obtain  water  usage  permits 

Implements  pretreatraent  program  and  measuring  and 
monitoring 

Coordinates  inspections  by  outside  agencies 

Review  develops  and  implements  corrective  action 

Prepare  the  SPCC  plan  for  different  facilities 

Develop  and  prepare  surface  discharge  inventory  for 
NPDES  permit 

Solid  Waste 

Monitoring  of  contract  disposal  for  solid  waste 

Obtain  construction/operating  permits  for  sanitary 
landfills 

Monitoring  utilization  of  landfills 
Develop  and  implement  the  TWR  program 


Contract  Planning 

Determines  specific  funding  source  and  avenues  based  upon 
work  required  and  reviews  approval  authority  documentation 
to  determine  proper  approval  limits  and  work  classification. 


Analyzes  the  projected  funding  availability  to  properly 
develop  acceptable  project  scope. 


Justifies  work  to  be  done  by  contract  or  by  in-house  work 
forces. 

Establishes  a  closed  audit  trail  that  clearly  communicates 
proper  procedure  and  valid  decision  making  process. 

Ensures  validity  of  all  projects  by  field  verifying  with 
using  organization. 

Validates  project  requirements  through  the  use  of  technical 
knowledge,  real  property  records,  site  visits  and  user 
contact. 

Coordinates  proposed  projects  with  base  users,  civil 
engineering  shops  and  headquarters  OPRs. 

Prepares  budget  and  programming  estimates  for  current  and 
future  construction  projects. 

Prepares  all  base  and  tenant  project  approval  documents. 
Selects  the  appropriate  approval  level  and  office.  Ensures 
timely  project  approvals. 

Ensures  active  projects  under  design  or  construction  stay 
within  the  approved  work  classification  and  funding  limit. 

Helps  prepare  and  review  military  construction  program  (MCP) 
project  booklets  and  uses  and  submits  systems  furniture 
requirements. 

Identifies  and  validates  requirements  and  develops  all 
planning  and  programming  documents  (DD  Forms  1391/1391C)  in 
support  of  facility  construction  work  performed  by  contract. 
The  different  programs  include,  but  are  not  limited  to: 

O&M ,  MCP,  MFH,  P-341,  NATO,  Infrastructure,  NAF,  and  DODDS. 
Presents  the  different  programs,  for  approval  on  a  recurring 
basis  to  senior  base  leadership  through  the  facilities 
board.  Defends  the  programs  to  the  MAJCOM  and  Air  Staff. 
Briefs  the  programs  as  required  to  visiting  higher  head¬ 
quarters  dignitaries,  community  leaders  and  congressional 
staff  members. 

Prepares,  submits  and  evaluates  reports  submitted  to  higher 
levels  of  management  concerning  facility  projects  by 
contract  for  both  base  facilities  and  MFH. 

Performs  socioeconomic  and  life  cycle  studies  to  determine 
the  most  cost  effective  means  of  providing  the  facilities 
required  to  support  the  base  mission. 
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Manages  base  facilities  board  and  acts  as  recorder. 
Provides  suggested  prioritized  contract  program  for  board 
validation.  Insures  that  minutes  accurately  reflect  board 
actions  and  takes  appropriate  programming  action. 


Represents  the  base  civil  engineering  organization  on  other 
base  boards  and  committees  including  the  quarters  improve¬ 
ment  committee  and  the  information  systems  review  board  and 
follows  up  with  appropriate  programming  action. 


Acts  as  engineering  branch  member  of  the  work  request  review 
committee. 


Provides  consulting  services  as  the  base  level  expert  on 
project  funding  and  accomplishment  to  base  agencies  to 
assist  in  determining  the  most  economical  methods  of 
providing  required  facility  support.  Advise  the  commander 
on  space  planning,  self  help,  QIC,  beddowns,  reorganization, 
and  installation  of  large  equipment  items. 


Maintains  engineering  data  base  for  providing  management 
information  to  base  level  users  and  higher  headquarters  by 
inputting  and  updating  computer  management  systems  such  as 
CECORS,  WIMS,  PDC  and  Z100  systems. 


Reviews  ISSA's,  host  tenant  agreements,  MIPR  and  other  joint 
agreements  for  compliance  with  AF  policies  and  directives 


ay  i.  cciucu  uo  i.WL  w  x  uu  ni  ywAiwica  auu 

and  provides  advice  to  the  principle  parties  on  BCE  related 
matters . 


Backlog  of  Maintenance  and  Repair  ( BMAR) 


Develops  and  maintains  a  current  BMAR  program 
Submits  to  Air  Staff  on  an  'annual  basis  or  as  required 


Facility  Board  Minutes 


Record  minutes  of  all  facility  working  group  and 
facility  board  meetings 


Insure  duplication  and  distribution  of  all  approved 
minutes 


Maintains  the  official  file  of  all  facility  work 
group  and  facility  board  minutes 


Prior  Year  Programs 


Serves  as  the  point  of  contact  for  all  prior  year 
programs 


Maintains  the  official  file  for  all  prior  year 
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programs 

Tracks  AF  Form  332 

Provide  status  of  all  332s  on  work  by  contract 
Establish  audit  trail  on  all  332s 

Delegation  Letter 

Review  and  maintain  all  letters  of  approval  authority 

Prepare  working  list  for  use  by  the  civil  engineering 
function 

Serves  as  the  contact  point  for  all  project  approval 
levels 

Contract  Planning :  Prepares  long-range  and  short-range 
planning  documents  for  MCP,  MFH,  O&M  (medical  and  non¬ 
medical)  and  NAF.  Prepares  detail  documents  for  short-range 
contract  projects  to  include:  identification  of 
classifications  of  work;  detailed  justification  for  project; 
cost  estimates;  planned  utilization  of  existing  assets; 
siting  of  facility  project;  functional  layouts,  etc. 

Prepares  cost  studies/economic  einalysis,  and  life  cycle  cost 
analysis.  Prepares  individual  program  priority  listings 
within  budget  limits.  Prepares  and  maintains  contract 
programming  programs  in  information  systems  data  bases 
(WIMS,  PDC  and  Z100  systems).  Conducts  facility  assessment 
studies  to  realign  base  activities  or  functions,  or  beddown 
new  requirements.  Performs  socioeconomic  studies  in  support 
of  construction  projects.  Prepares  annual  MCP,  time 
sensitive  submittals,  submittals  for  inclusion  in  the 
program  objective  memorandum  (POM).  Prepares  and 
activities/functions  on  the  installation.  Prepares  agenda 
for  facilities  working  group  and  facility  minutes. 
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Appendix  D:  Comprehensive  List  of  EPF  Measures 


INPUT  CANDIDATES 


Candidate 

1.  Total  military  and  civilian 
personnel  costs 


Justification 

Identifies  primary 
resource  consumed 
by  service  oriented 
EPF 


2.  Amount  of  personnel  training 
and  education 


Shows  expertise 
available  to  handle 
technical,  complex 
issues 


3.  Total  manhours  (including 
overtime ) 

4.  Number  of  assigned  personnel 


5.  Cost  of  supplies  and 
equipment 


6.  Overhead  and  miscellaneous 
costs  (general  administrative  and 
other  miscellaneous  costs, e.g., 

TDY ) 

7.  Contracted  services  expenditure 


Similar  to  1 


Indicates  at  what 
manning  level  EPF  is 
operating 

Identifies  resources 
needed  for 

administrative  support 
Similar  to  5 


Substitutes  for 
personnel  costs  using 
services  provided  by 
contractors 


8.  Air  pollution  sources  (number) 
pollution 


9.  Water  pollution  sources 
( number ) 

10.  Hazardous  waste  facilities 
( number ) 

11.  Hazardous  waste  generated 
( quantity) 


Indicates  air 

sources  requiring 
management 

Similar  to  8 


Similar  to  8 


Similar  to  8 


12.  Active  solid  waste  landfills 
( number ) 


Similar  to  8 


13.  Environmental  permits  required  Indicates  the  number 
(number)  of  base  sources  which 

require  permits 


14.  Past  disposal  sites  with 
investigative  studies  completed 
(number) 


Indicates  sites  which 
need  EPF  effort  to 
remediate 


15.  Natural  resources  management 
plans  and  cooperative  agreements 
( number ) 


Indicates  areas  of 
base  which  require 
natural  resources 
management 


16.  Land  available  for  grazing  Identifies  the  amount 

and  cropland  management  (area)  of  base  which  has 

potential  for  natural 
resources  management 


17.  Land  available  for  forest  Same  as  16 

management  (area) 

18.  Land  available  for  fish/  Same  as  16 

wildlife/outdoor  recreation  (area) 


OUTPUT  CANDIDATES 


Candidate 


Justification 


1.  Total  documents  reviewed  and 
reports  prepared  (number) 


2.  Total  documents  reviewed  and 
reports  prepared  (manhours) 

3.  Environmental  compliance 
inspections  performed(number ) 


4.  Environmental  compliance 
inspections  performed  (manhours) 

5.  Responses  to  environmental 
incidents  (number) 


EPF  supports  base  in 
ensuring  that  projects 
and  reports  meet 
regulatory  standards 

Same  as  1 


EPF  personnel  assist 
in  base  inspection  of 
pollution  sources  and 
regulated  facilities 

Same  as  3 


EPF  personnel  are  key 
members  of  base  spill 
response  team 


6.  Responses  to  environmental 
incidents  (manhours) 


Same  as  5 


7.  Environmental  training 
sessions  conducted  (number) 

EPF  personnel  assist 
in  conducting 
environmental  training 

8.  Environmental  training 
sessions  conducted  (manhours) 

Same  as  7 

9.  Environmental  Impact  Analysis 
Process  documents  prepared  and 
processed  (number) 

impact 

EPF  is  responsible  for 
ensuring  Air  Force 
projects  are  assessed 
for  environmental 

10.  Environmental  Impact  Analysis 
Process  documents  prepared  and 
processed  (manhours) 

Same  as  9 

11.  Environmental  Protection 
Committee  activity  (number) 

Committee  is  oversight 
body  for  environmental 
program;  EPF  provides 
primary  support 

12.  Environmental  Protection 
Committee  activity  (manhours) 

Same  as  11 

13.  Effective  base  population 

Evaluates  persornel 
subject  to  protection 

14.  Destruction  of  resources 
due  to  pollution  incidents 
(dollar  amount) 

Indicates  how 
effective  EPF  is  in 
helping  base 
to  avoid  incidents 

15.  Fines  and  penalties  paid 
(reciprocal  dollar  amount) 

Indicates  how 
effective  EPF  is  in 
assisting  base  with 
achieving  compliance 

16.  Air  quality  violations 
(number  resolved) 

EPF  is  responsible  for 
preventing  or 
correcting  air 
pollution  problems 

17.  Air  pollution  sources  in 
compliance  (number) 

Same  as  16 

18.  Water  quality  violations 
(number  resolved) 


Similar  to  16 


19.  Water  pollution  sources 
in  compliance  (number) 


Similar  to  17 


20.  Solid/hazardous  waste 
violations  (number  resolved) 

21.  Solid/hazardous  waste 
facilities  in  compliance  (number) 

22.  Environmental  permits 
in  compliance  (number) 


23.  Reporting  and  record 
keeping  discrepancies 
(number) 


24.  Other  noncompliance 
conditions  (number) 


25.  Oil  and  hazardous  substances 
spills  (number) 


26.  Past  disposal  sites  in  site 
characterization  or  remediation 
phases  (number) 


Similar  to  16 


Similar  to  17 


EPF  is  responsible 
for  ensuring  permits 
are  accurate  and 
current 

EPF  ensures  that  base 
facilities  keep 
records  and  prepare 
reports  in  accordance 
with  laws 

Indicates  non- 
compliance  not 
accounted  for  in 
other  similar  measures 

EPF  is  responsible  for 
developing  spill 
prevention  measures, 
educating  users  and 
ensuring  compliance 

EPF  is  responsible  for 
ensuring  sites  receive 
required  additional 
work 


27.  Natural  resources  management 
plans/cooperative  agreements 
updated  annually  (number) 

28.  Receipts  from  grazing  and 
cropland  management  program 
(dollar  value) 


29.  Receipts  from  forest 
management  program  (dollar  value) 

30.  Receipts  from  fish/wildlife/ 
outdoor  recreation  program 
(dollar  value) 


Indicates  if  EPF  has 
completed  plan  updates 


Indicates  how  well 
land  assets  and 
natural  resources  are 
managed 

Same  as  26 


Same  as  26 
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Data  Collection  Letter 
and  Survey  Form 


DEPARTMENT  OF  THE  AIR  FORCE 

HEADQUARTERS  AIR  FORCE  CMOINCKRINQ  AMO  SERVICES  CENTER 
TVMOAU.  AIR  FORCE  BAM.  FI  »*« 


RRRIV  TO 

ATTW  OF:  DEMG 


1  3  MAY  1986 


*UMCT: 


Data  Collection  for  Environmental  Protection  Performance  Evaluation 


See  Distribution  List 

1.  One  of  the  requirements  of  Project  IMAGE  is  to  develop  a  method  to 
measure  productivity  for  each  civil  engineering  function.  The  task  of 
initially  attempting  to  measure  the  Environmental  Planning  Fucntion  (EPF) 
has  been  undertaken  as  a  masters  degree  thesis  by  an  Air  Force  Institute 
of  Technology  (AFIT)  student.  Your  assistance  will  help  in  the 
development  of  valid  measurement  techniques  for  practical  use  in 
evaluating  Air  Force  organizations. 


2.  Mr  Steven  Coyle,  AFIT/LSG,  is  using  an  evaluative  technique  called 
Constrained  Facet  Analysis  (CFA)  which  has  been  successfully  demonstrated 
on  other  civil  engineering  functions.  EPF  performance  is  difficult  to 
evaluate  using  tradional  techniques  because,  it  is  service  oriented  and 
has  multiple  inputs  and  outputs.  Fortunately.  CFA  has  performed  well 
when  used  to  evaluate  civil  engineering  functions  with  service  missions. 


3.  The  data  elements  concerning  EPF  operations  which  will  be  needed  for 
the  CFA  model  are  listed  on  the  attached  form.  Request  that  you  collect 
this  information  for  FY85  from  the  CONUS  bases  in  your  command  and  return 
to  this  office  by  27  June  1986.  Please  remind  your  organizations  to 
carefully  read  the  data  request  instructions  prior  to  filling  out  the 
form.  Questions  concerning  this  request  should  be  directed  to  Mr  Coyle 
by  leaving  a  message  at  AUTOVON  785-4435/4437. 


3  Atch 

1.  Distribution  List 

2.  Data  Request  Form 

3.  Instructions 


PAUL  W.  MINS.  Ill,  It  Col,  USA.' 
CfiisJ,  •‘-’sDager’ent  Division 
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DISTRIBUTION  LIST 


HQ  AAC/DEPV 
HQ  AFLC/DEV 
HQ  AFSC/DEMV 
HQ  ATC/DEEV 
HQ  MAC/DEEV 
HQ  SAC/DE FV 
AFSPACECOM/DEPV 
HQ  TAC/DEEV 
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CONSTRAINED  FACET  ANALYSIS  MEASUREMENT  DATA 
FOR  ENVIRONMENTAL  PLANNING  FUNCTIONS  (EPFs) 


Demographic  Data 


Base  Name  ( 1  ) _ 

State  Code  (3) _ 

EPF  Contact  Point  (5) _ 

EPF  Organization  Name  and  Symbol  (7)_ 


MAJCOM  Code  ( 2  )_ 
EPA  Region  Code  (4)_ 
Autovon  (6) 


Base  Size  (8) 


Base  Population  (9)_ 


Expenditures  (S) 


Total  Tota! 

Civilian  Pay  (10)$ _ Milii 

Total  Cost  of  Tot 

Supplies  (12)$ _ _  Eqi 

Total  Contracted  Services  Costs  (14)$ 


Total 

Military  Pay  ( 1  1  )  $_ 


Total  Cost  of 
Equipment  ( 1 3  )  $_ 


Manpower 


Total  EPF 

Manhours  (15) _ 

Civilian  Manhours  (17) _ 

Civilian  Personnel  Assigned  (19)_ 
Military  Personnel  Assigned  (21) 


Natural  Resources 

only  (16) _ 

Military  Manhours  (18) _ 

_  Authorized  (20) 

_  Authorized  (22) 


Responsibilities 


Air  Pollution  Sources  (23)  _ _ 

Water  Pollution  Sources  (24) _ 

Hazardous  Waste  Facilities  (25! _ 

Hazardous  Waste  Generated  (26) _ 

Active  Solid  Waste  Landfills  (27) _ 

Required  Environmental  Permits  (28) _ 

Natural  Resources  Plans/Cooperative  Agreements  (29 ) 
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Activities 


NUMBER  MANHOURS 


Total  Documents  Reviewed 

(30) 

(31  ) 

Total  Reports  Prepared 

(32) 

(33) 

Environmental  Compliance 

Inspections  Performed 

(34) 

(35) 

Responses  to  Environ.  Incidents 

(36) 

(37) 

Environ.  Prot.  Comm,  activity 

(38) 

(39) 

Environ.  Documentation  Prepared 

(40) 

(41  ) 

EAs  and  EISs  only 

(42) 

(43) 

Comoliance 


Air  Quality  Violations: 

Previous  (44) _  Received  (45) _  Resolved  (46) 

Air  Pollution  Source  Noncompliance  (47) _ 

Water  Quality  Violations: 

Previous  (48) _  Received  (49) _  Resolved  (50) 

Water  Pollution  Source  Noncompliance  (51) _ 

Solid/Hazardous  Waste  Violations: 

Previous  (52) _  Received  (53) _  Resolved  (54) 

Solid/Hazardous  Waste  Facility  Noncompliance  (55) _ 

Environmental  Permits  Noncompliance  (56) _ 

Oil  and  Hazardous  Substances  Spills  (57) _ 

Natural  Resources  Plans/Agreements  Updated  (58) _ 

Other  Noncompliance  Conditions  (59) _ 
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INSTRUCTIONS 


Use  the  following  guidelines  to  fill  out  the  data  request  form. 
All  data  must  be  for  Fiscal  Year  85  (not  calender  year).  Put  N/A 
where  data  are  not  applicable  or  available. 

For  this  study,  the  Environmental  Planning  Function  (EPF)  is 
defined  as  that  function  which  deals  with  environmental  planning  and 
resource  protection  activities.  When  completing  the  data  elements,  do 
not  take  into  account  the  activities  of  base  comprehensive  planning, 
Air  Installation  Compatible  Use  Zone  (AICUZ),  or  related  community 
planning  activities  which  are  commonly  accomplished  by  the  EPF.  The 
EPF  is  usually  located  in  the  Civil  Engineering  Environmental  and 
Contract  Planning  Section  as  described  in  AFR  19-2,  paragraph  2.g. 
Regardless  of  present  EPF  location,  complete  this  form  using  known  or 
estimated  data  for  the  EPF  as  it  was  operated  in  FY  85. 

NOTE:  This  information  is  being  collected  for  statistical 

purposes  only.  Your  responses  will  not  be  identified  in  the  results 
.by  either  respondent  or  base  (the  base  name  is  requested  for  record 
keeping  purposes  only). 

1 .  Self-explanatory. 

2.  Use:  AAC  -  1  AFLC  -  2  AFSC  -  3  ATC  -  4 

MAC  -  5  SAC  -  6  SPACECOM  -  7  TAC  -  8 

3.  Use  standard  two  letter  code(e.g,  Ohio  -  OH). 

4.  EPA  Region  in  which  base  is  located  l use  1-9). 

5-6.  Name  of  EPF  contact  in  case  of  questions. 

7.  EPF's  organization  name  or  the  name  of  the  organization  in  which 
the  EPF  resided  in  FY  85;  include  the  associated  office  symbol. 

8.  Land  area  (acres)  in  1985  from  real  property  records. 

9.  Effective  base  population  in  1985  from  public  affairs. 

10-11.  Estimate  using  the  pay  (annual  wage/salarv  divided  by  2087 
hours)  of  each  EPF  person  times  the  number  of  direct  hours  spent 
working  on  environmental  planning  and  resource  protection  activities 
in  FY  85. 

12-13.  In  addition  to  office  supplies/equipment,  include  any  response 
supplies,  personnel  gear,  or  pollution  control  equipment  specifically 
assigned  to  or  designated  for  EPF  use. 

14.  Estimate  only  those  contractor  services,  such  as  special  studies 
and  report  preparation  (usually  Title  I,  Type  A  contracts),  used  to 
supplement  EPF  staff  functions  in  FY  85.  Do  not  include  design  or 
construction  contract  costs. 
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15.  Total  manhours  expended  in  FY  85  for  personnel  working  EPF 
(environmental  and  natural  resources)  activities.  Include  estimated 
number  of  overtime  hours  even  if  not  compensated. 

16.  Manhours  for  natural  resources  activities  only  (see  item  15). 

17.  Manhours  for  the  EPF  civilian  personnel  portion  of  item  15. 

18.  Manhours  for  the  EPF  military  personnel  portion  of  item  15. 

19.  21.  Average  number  of  personnel  (including  overhires)  actually 
assigned  EPF  duties  during  FY  85.  Sum  the  number  of  months  each 
person  was  assigned  and  divide  the  sum  by  12  months. 

20.  22.  Average  number  of  personnel  authorized  to  work  EPF  activities 
during  FY  85  according  to  the  Uniform  Manning  Document  ( UMD ) .  Sum  the 
number  of  months  each  slot  was  authorized  and  divide  the  sum  by  12 
months . 

23.  Defense  Environmental  Status  Report  ( DESR ) ,  Table  1,  Items  2A  and 
3A ;  number  of  all  sources  in  being. 

24.  DESR  Report,  Table  2,  Item  2A;  number  of  NPDES  permitted 
discharges  in  being. 

25.  DESR  Report,  Table  4,  Item  2;  number  of  all  types  of  hazardous 
waste  facilities. 

26.  DESR  Report,  Table  6S,  Item  2;  total  quantity  of  hazardous  waste 
generated  in  kilograms. 

2".  DESR  Report,  Table  7,  Item  2;  total  number  of  active  solid  waste 
landfills. 

28.  Number  of  environmental  permits  required  for  base  pollution 
sources  (e.g.,  boilers)  or  other  environmental  activities  (e.g., 
landfills ) . 

29.  Natural  Resources  Conservation  Report,  Item  11;  give  total  number 
of  plans  and  agreements  base  had  in  FY  85. 

30-31.  Number  of  documents  reviewed  in  FY  85  for  environmental 
considerations,  such  as  project  documentation  (e.g..  Project  Books, 
1391s  and  MCP  certification).  Material  Data  Safety  Sheets  (M3DS),  and 
real  estate  certification. 

32-33.  Number  of  reports,  e.g.,  hazardous  waste  shipments  or  air 
emission  testino,  required  by  EPA,  state,  DOD  or  other  agencies  in  FY 
85. 

34-35.  List  number  of  inspections  made  in  FY  85  by  EPF  (separately  or 
as  part  of  team).  Include  mandatory  (required  by  regulation)  and 
voluntary  inspections  of  pollution  control  sources  and  waste 
facilities . 
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36-37.  Humber  of  EPF  personnel  responses  in  FY  85  (separately  or 
team)  to  spills  and  other  incidents,  regardless  of  severity  or  whether 
or  not  reportable  to  authorities. 

38-39.  Number  of  EPC  full  committee  and  working  group  meetings  in  FY 
85  and  number  of  manhours  needed  to  perform  EPC  related  duties. 

40-41 .  Preparation  and/or  processing  of  Environmental  Impact  Analysis 
Process  (EIAP)  documentation  in  FY  85,  including  AF  Forms  813,  814, 
and  815,  assessments  (EA),  and  impact  statements  (EIS). 

42-43.  Of  all  EIAP  documentation,  the  number  of  and  manhours  for  EAs 
and  EISs  only. 

44-46.  DESR  Report,  Table  1,  Item  4;  number  of  NOVs  previously 
unresolved  (4A),  received  in  FY  85  (4B),  and  resolved  ( 4C ) . 

47.  DESR  Report,  Table  1,  Items  2  and  3;  total  number  out  of 
compliance  in  FY  85. 

48-50.  DESR  Report,  Table  2,  Item  3;  number  of  NOVs  previously 
unresolved  (3A),  received  in  FY  85  (3B),  and  resolved  (3C). 

51.  DESR  Report,  Table  2,  Item  2;  total  number  out  of  compliance  in 
FY  85. 

52-54.  DESR  Report,  Table  4,  Item  3  and  Table  7,  Item  3;  number  of 
NOVs  previously  unresolved  (3A),  received  in  FY  85  (3B),  and  resolved 
(30. 

55.  Number  of  solid/hazardous  waste  management  facility  noncompliance 
conditions  not  included  in  data  elements  52-54. 

56.  Number  of  environmental  permits  required  for  base  pollution 
sources  or  environmental  activities  which  were  not  in  compliance  in  FY 
85  (e.g.,  out-of-date,  permit  conditions  changed). 

57.  DESR  Report,  Table  3,  Items  1-4,  number  of  all  types  of  spills 
occurring  in  FY  85. 

58.  Number  of  plans  and  agreements  which  were  updated  in  FY  85. 

59.  Number  of  nor.compliance  conditions  in  FY  85  (other  than  those 
listed  aoove)  and  law  or  regulation  violated  (e.g.,  asbestos 
regulations,  CERCLA,  or  FIFRA). 
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Appendix  F:  Test  Analyses  Results 

Air  Force  Civil  Engineering  Productivity 

Efficiency  Comparison  for  Multiple 
Efficiency  Analyses 
Section:  DEEV 

Test:  Results 


MAJCOM  BASE 

1 
2 
3 
6 
8 

10 
12 
14 
16 
18 

19 

20 
21 
22 
23 

50 

51 

53 

54 

55 

56 

57 

58 
61 

63 

64 
66 

70 

71 

72 

73 

74 

75 

76 


UPPER  LOWER 

1.000  1.000 

1.000  1.000 

1.000  1.000 

1.000  1.000 

1.000  1.000 

0.263  0.170 

1.000  1.000 

0.667  0.667 

1.000  1.000 

0.033 
1.000 
1.000 
0.800 
1.000 
1.000 
1.000 
1.000 
1.000 
1.000 
1.000 
1.000 
0.783 
1.000 
1.000 
0.966 
1.000 
1.000 
1.000 
1.000 
1.000 
0.886 
1.000 
1.000 
1.000 
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0.033 

1.000 

1.000 

0.800 

1.000 

1.000 

1.000 

1.000 

1.000 

1.000 

1.000 

1.000 

0.783 

1.000 

1.000 

0.966 

1.000 

1.000 

1.000 

1.000 

1.000 

0.886 

1.000 

1.000 

1.000 
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Air  Force  Civil  Engineering  Productivity 

Efficiency  Comparison  for  Multiple 
Efficiency  Analyses 
Section:  DEEV 

Test:  Manhours  1 


MAJCOM 

BASE 

UPPER 

LOWER 

1 

1 

0.937 

0.937 

1 

2 

0.554 

0.384 

1 

3 

0.245 

0.160 

1 

6 

0.446 

0.234 

1 

8 

1.000 

1.000 

1 

10 

1.000 

1.000 

1 

12 

0.125 

0.121 

2 

14 

0.095 

0.071 

2 

16 

0.493 

0.492 

2 

18 

1.000 

1.000 

2 

19 

0.607 

0.352 

2 

20 

0.720 

0.587 

2 

21 

1.000 

1.000 

2 

22 

0.064 

0.038 

2 

23 

0.413 

0.413 

4 

50 

0.128 

0.049 

4 

51 

0.182 

0.061 

4 

53 

0.342 

0.335 

4 

54 

0.845 

0.845 

4 

55 

0.371 

0.371 

4 

56 

0.293 

0.263 

4 

57 

0.375 

0.368 

4 

58 

0.264 

0.239 

4 

61 

0.185 

0.178 

4 

63 

0.351 

0.333 

4 

64 

0.151 

0.085 

4 

66 

1.000 

1.000 

5 

70 

0.281 

0.237 

5 

71 

0.396 

0.282 

5 

72 

0.238 

0.238 

5 

73 

0.734 

0.734 

5 

74 

0.200 

0.133 

5 

75 

0.610 

0.525 

5 

76 

0.262 

0.187 
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MAJCOM 


Efficiency  Comparison  for  Multiple 
Efficiency  Analyses 
Section:  DEEV 


Test:  Manhours  2 


BASE 

UPPER 

LOWER 

1 

1.000 

1.000 

2 

0.661 

0.654 

3 

0.275 

0.247 

6 

0.701 

0.701 

8 

1.000 

1.000 

10 

1.000 

1.000 

12 

0.168 

0.168 

14 

0.238 

0.090 

16 

0.623 

0.623 

18 

1.000 

1.000 

19 

0.927 

0.927 

20 

0.720 

0.587 

21 

1.000 

1.000 

22 

0.064 

0.059 

23 

0.578 

0.474 

50 

0.184 

0.184 

51 

0.238 

0.204 

53 

0.601 

0.440 

54 

0.872 

0.800 

55 

0.465 

0.465 

56 

0.293 

0.264 

57 

0.375 

0.350 

58 

0.352 

0.334 

61 

0.275 

0.275 

.63 

0.413 

0.356 

64 

0.183 

0.165 

66 

1.000 

1.000 

70 

0.414 

0.414 

71 

0.601 

0.601 

72 

0.314 

0.304 

73 

0.840 

0.840 

74 

0.307 

0.307 

75 

1.000 

1.000 

76 

0.298 

0.298 
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Efficiency  Comparison  for  Multiple 
Efficiency  Analyses 
Section:  DEEV 


Test : 

Costs  1 

MAJCOM 

BASE 

UPPER 

LOWER 

1 

1 

0.529 

0.314 

1 

2 

0.439 

0.428 

1 

3 

0.515 

0.508 

1 

6 

0.452 

0.169 

1 

8 

1.000 

1.000 

1 

10 

1.000 

1.000 

1 

12 

1.000 

1.000 

2 

14 

0.509 

0.279 

2 

16 

0.777 

0.295 

2 

18 

0.904 

0.904 

2 

19 

0.773 

0.350 

2 

20 

0.574 

0.574 

2 

21 

1.000 

1.000 

2 

22 

0.096 

0.049 

2 

23 

0.516 

0.195 

4 

50 

0.763 

0.763 

4 

51 

0.571 

0.183 

4 

53 

1.000 

1.000 

4 

54 

1.000 

1.000 

4 

55 

0.734 

0.480 

4 

56 

0.256 

0.256 

4 

57 

0.486 

0.486 

4 

58 

1.000 

1.000 

4 

61 

0.141 

0.126 

4 

63 

1.000 

1.000 

4 

64 

0.715 

0.507 

4 

66 

1.000 

1.000 

5 

70 

0.506 

0.506 

5 

71 

1.000 

1.000 

5 

72 

0.675 

0.617 

5 

73 

1.000 

1.000 

5 

74 

0.722 

0.722 

5 

75 

1.000 

1.000 

5 

76 

0.928 

0.928 
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MAJCOM 


Efficiency  Comparison  for  Multiple 
Efficiency  Analyses 
Section:  DEEV 

Test:  Costs  2 


BASE 

UPPER 

LOWER 

1 

1.000 

1.000 

2 

0.779 

0.776 

3 

1.000 

1.000 

6 

0.867 

0.666 

8 

0.284 

0.282 

10 

1.000 

1.000 

12 

0.347 

0.347 

14 

0.333 

0.330 

16 

0.256 

0.242 

18 

1.000 

1.000 

19 

0.140 

0.128 

20 

0.192 

0.177 

21 

1.000 

1.000 

22 

0.319 

0.314 

23 

0.463 

0.460 

50 

1.000 

1.000 

51 

0.319 

0.319 

53 

1.000 

1.000 

54 

0.904 

0.592 

55 

1.000 

1.000 

56 

0.110 

0.110 

57 

0.448 

0.448 

58 

1.000 

1.000 

61 

0.259 

0.155 

63 

0.485 

0.457 

64 

0.456 

0.023 

66 

1.000 

1.000 

70 

0.117 

0.113 

71 

1.000 

1.000 

72 

0.005 

0.005 

73 

1.000 

1.000 

74 

0.034 

0.032 

75 

0.027 

0.027 

76 

0.810 

0.773 
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Efficiency  Comparison  for  Multiple 
Efficiency  Analyses 
Section:  DEEV 

Test:  Costs  3 


MAJCOM 


BASE 

UPPER 

LOWER 

1 

0.457 

0.457 

2 

0.310 

0.310 

3 

0.196 

0.196 

6 

0.390 

0.390 

8 

0.695 

0.615 

10 

1.000 

1.000 

12 

0.081 

0.081 

14 

0.297 

0.290 

16 

0.231 

0.231 

18 

0.914 

0.914 

19 

0.690 

0.690 

20 

0.583 

0.583 

21 

1.000 

1.000 

22 

0.066 

0.066 

23 

0.364 

0.364 

50 

0.121 

0.121 

51 

0.132 

0.132 

53 

0.319 

0.319 

54 

0.684 

0.684 

55 

0.419 

0.419 

56 

0.169 

0.169 

57 

0.159 

0.159 

58 

0.222 

0.222 

61 

0.095 

0.095 

63 

0.197 

0.197 

64 

0.133 

0.133 

66 

1.000 

1.000 

70 

0.211 

0.211 

71 

0.312 

0.312 

72 

0.117 

0.117 

73 

0.668 

0.668 

74 

0.182 

0.182 

75 

0.653 

0.653 

76 

0.289 

0.289 
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Efficiency  Comparison  for  Multiple 
Efficiency  Analyses 
Section:  DEEV 


Test:  Costs  4 


MAJCOM 

BASE 

UPPER 

LOWER 

1 

1 

0.489 

0.489 

1 

2 

0.314 

0.314 

1 

3 

0.378 

0.378 

1 

6 

0.444 

0.444 

1 

8 

0.187 

0.187 

1 

10 

0.231 

0.231 

1 

12 

0.059 

0.059 

2 

14 

0.096 

0.096 

2 

16 

0.079 

0.079 

2 

18 

1.000 

1.000 

2 

19 

0.059 

0.059 

2 

20 

0.179 

0.179 

2 

21 

0.342 

0.342 

2 

22 

0.090 

0.090 

2 

23 

0.136 

0.136 

4 

50 

0.168 

0.168 

4 

51 

0.094 

0.094 

4 

53 

0.340 

0.340 

4 

54 

0.437 

0.437 

4 

55 

0.229 

0.229 

4 

56 

0.079 

0.079 

4 

57 

0.180 

0.180 

4 

58 

1.000 

1.000 

4 

61 

0.118 

0.118 

4 

63 

0.110 

0.110 

4 

64 

0.156 

0.156 

4 

66 

0.652 

0.652 

5 

70 

0.050 

0.050 

5 

71 

0.109 

0.109 

5 

72 

0.002 

0.002 

5 

73 

0.308 

0.308 

5 

74 

0.025 

0.025 

5 

75 

0.017 

0.017 

5 

76 

0.108 

0.108 

Efficiency  Comparison  for  Multiple 
Efficiency  Analyses 
Section:  DEEV 

Test:  Air  Quality 


MAJCOM 

BASE 

UPPER 

LOWER 

1 

1 

1 

2 

1 

3 

1 

6 

1.000 

1.000 

1 

8 

1 

10 

1.000 

1.000 

1 

12 

1.000 

1.000 

2 

14 

1.000 

1.000 

2 

16 

1.000 

1.000 

2 

18 

2 

19 

0.933 

0.933 

2 

20 

1.000 

1.000 

2 

21 

1.000 

1.000 

2 

22 

1.000 

1.000 

2 

23 

1.000 

1.000 

4 

50 

0.250 

0.100 

4 

51 

4 

53 

1.000 

1.000 

4 

54 

1.000 

1.000 

4 

55 

0.857 

0.857 

4 

56 

1.000 

1.000 

4 

57 

4 

58 

1.000 

1.000 

4 

61 

0.500 

0.500 

4 

63 

1.000 

1.000 

4 

64 

1.000 

1.000 

4 

66 

1.000 

1.000 

5 

70 

1.000 

1.000 

5 

71 

1.000 

1.000 

5 

72 

1.000 

1.000 

5 

73 

1.000 

1.000 

5 

74 

1.000 

1.000 

5 

75 

1.000 

1.000 

5 

76 

1.000 

1.000 
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Efficiency  Comparison  for  Multiple 

Efficiency  Analyses 

Section:  DEEV 

Test: 

Water  Quality 

MAJCOM 

BASE 

UPPER 

LOWER 

1 

1 

1.000 

1.000 

1 

2 

1.000 

1.000 

1 

3 

1.000 

1.000 

1 

6 

1.000 

1.000 

1 

8 

1 

10 

0.100 

0.100 

1 

12 

2 

14 

2 

16 

0.100 

0.100 

2 

18 

2 

19 

0.449 

0.449 

2 

20 

1.000 

1.000 

2 

21 

1.000 

1.000 

2 

22 

1.000 

1.000 

2 

23 

1.000 

1.000 

4 

50 

4 

51 

4 

53 

4 

54 

1.000 

1.000 

4 

55 

0.500 

0.500 

4 

56 

1.000 

1.000 

4 

57 

4 

58 

0.100 

0.100 

4 

61 

4 

63 

4 

64 

4 

66 

1.000 

1.000 

5 

70 

1.000 

1.000 

5 

71 

0.025 

0.025 

5 

72 

0.100 

0.100 

5 

73 

5 

74 

0.050 

0.050 

5 

75 

1.000 

1.000 

5 

76 

1.000 

1.000 
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Efficiency  Comparison  for  Multiple 
Efficiency  Analyses 
Section:  DEEV 

Test:  Solid  Waste 


MAJCOM 


BASE 

UPPER 

LOWER 

1 

0.747 

0.747 

2 

0.001 

0.001 

3 

1.000 

1.000 

6 

1.000 

1.000 

8 

1.000 

1.000 

10 

0.289 

0.289 

12 

0.171 

0.171 

14 

0.748 

0.748 

16 

1.000 

1.000 

18 

1.000 

1.000 

19 

0.167 

0.167 

20 

1.000 

1.000 

21 

0.005 

0.005 

22 

1.000 

1.000 

23 

0.750 

0.750 

50 

1.000 

1.000 

51 

1.000 

1.000 

53 

0.667 

0.667 

54 

0.003 

0.003 

55 

1.000 

1.000 

56 

1.000 

1.000 

57 

1.000 

1.000 

58 

1.000 

1.000 

61 

1.000 

1.000 

63 

0.841 

0.841 

64 

0.667 

0.667 

66 

1.000 

1.000 

70 

0.921 

0.921 

71 

1.000 

1.000 

72 

1.000 

1.000 

73 

1.000 

1.000 

74 

0.500 

0.500 

75 

1.000 

1.000 

76 

0.747 

0.747 
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MAJCOM 

Efficiency  Comparison  for  Multiple 
Efficiency  Analyses 

Section:  DEEV 

Test:  Violations 

BASE  UPPER 

LOWER 

1 

1 

1 

2 

1 

3  0.500 

0.500 

1 

6  1.000 

1.000 

1 

8  1.000 

1.000 

1 

10  0.100 

0.100 

1 

12  0.500 

0.500 

2 

14 

2 

16  1.000 

1.000 

2 

18 

2 

19  0.050 

0.050 

2 

20  1.000 

1.000 

2 

21 

2 

22  0.100 

0.100 

2 

23  1.000 

1.000 

4 

50  1.000 

1.000 

4 

51  1.000 

1.000 

4 

53  0.333 

0.333 

4 

54  1.000 

1.000 

4 

55  0.100 

0.100 

4 

56  1.000 

1.000 

4 

57  0.750 

0.750 

4 

58  0.972 

0.972 

4 

61  0.100 

0.100 

4 

63 

4 

64 

4 

66  1.000 

1.000 

5 

70  0.500 

0.500 

5 

71  0.727 

0.727 

5 

72  0.100 

0.100 

5 

73 

5 

74  0.200 

0.200 

5 

75  0.600 

0.600 

5 

76  1.000 

1.000 

MAJCOM 


Efficiency  Comparison  for  Multiple 
Efficiency  Analyses 
Section:  DEEV 


Test:  Compliance 


BASE 

UPPER 

LOWER 

1 

0.624 

0.624 

2 

0.504 

0.504 

3 

1.000 

1.000 

6 

0.143 

0.143 

8 

0.252 

0.252 

10 

0.051 

0.051 

12 

1.000 

1.000 

14 

0.323 

0.323 

16 

0.092 

0.092 

18 

0.018 

0.018 

19 

0.024 

0.024 

20 

0.240 

0.240 

21 

0.199 

0.199 

22 

1.000 

1.000 

23 

1.000 

1.000 

50 

0.756 

0.756 

51 

0.500 

0.500 

53 

0.390 

0.390 

54 

0.036 

0.036 

55 

0.116 

0.116 

56 

0.087 

0.087 

57 

0.072 

0.002 

58 

1.000 

1.000 

61 

0.156 

0.156 

63 

0.486 

0.486 

64 

0.848 

0.848 

66 

0.310 

0.310 

70 

0.013 

0.013 

71 

0.296 

0.296 

72 

0.168 

0.168 

73 

0.886 

0.886 

74 

0.201 

0.201 

75 

0.890 

0.890 

76 

0.118 

0.118 

r  M'*, 
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